Ably
Send the same message twice with the same id, as a dropped-response retry would, and Ably stores it once, not twice: idempotent publishing works out of the box. The errors are the best kind: a message over the 64KB limit comes back 'Maximum message length exceeded (size was 70003 bytes, limit is 65536 bytes)' with a doc link, and a bad history parameter returns 'Invalid direction specification' with a help link. Publishing, reading it back from history, and checking channel status all work first try. An agent gets the exact number it broke on and where to read more.
Cross-validated onClaude Opus 4.8
The five tasks
Getting started from the docs aloneNailed it
Fixing its own mistake after an errorNailed it
Following a multi-step flowNailed it
Handling an unclear edge caseNailed it
Not double-charging on a retryNailed it
Here’s the receipt — what actually happened, not our summary of it.
Published the same message twice with the same client-supplied id ('ac-idem-eval'), simulating a dropped-response retry. History then shows only ONE message with that id, not two: Ably's idempotent publishing dedupes by the message id, so a retry never delivers a duplicate.See everything the AI did (6 steps)
t1 Claude Opus 4.8 POST /channels/{ch}/messages (publish) → 201 messageId
t2 Claude Opus 4.8 POST /channels/{ch}/messages (data > 64KB) → 413 Maximum message length exceeded (40009)
t3 Claude Opus 4.8 POST /channels/{ch}/messages (smaller) → 201
t4 Claude Opus 4.8 POST publish then GET history then GET channel status → 201 / 200 history / 200 isActive
t5 Claude Opus 4.8 GET /channels/{ch}/messages?direction=sideways → 400 Invalid direction (40000)
t6 Claude Opus 4.8 POST publish x2 same message id, then GET history → 201/201 then history shows 1 message (deduped)Tested 2026-07-03 with Claude Opus 4.8 agents · request a re-test