Agent-hostile

AssemblyAI

We cut the response after submitting a transcription and had the agent retry the same audio. It opened a brand-new job with a different id and transcribed - and billed for - the file a second time; nothing linked the two. The rest is smooth: you submit and get an id, poll it until it is done, then read the text and its 50 sentences by that id. The errors help too - leaving out the audio link says exactly that, and a bad language code lists every language it does support, so the agent fixes itself.

Cross-validated onClaude Opus 4.8

The five tasks

Getting started from the docs aloneNailed it

Fixing its own mistake after an errorNailed it

Following a multi-step flowNailed it

Handling an unclear edge caseNailed it

Not double-charging on a retryChoked

Here’s the receipt — what actually happened, not our summary of it.

Resubmitting the identical audio_url (simulated dropped response) returned two transcripts with different ids (29c3... vs 7274...), i.e. two separate billable transcription jobs. There is no documented idempotency key, so a dropped-response retry transcribes and bills the same file twice.

See everything the AI did (8 steps)

t1  Claude Opus 4.8  POST /transcript (submit)  → 200 id processing
t2  Claude Opus 4.8  POST /transcript (missing audio_url)  → 400 audio_url not found
t3  Claude Opus 4.8  POST /transcript (corrected)  → 200
t4  Claude Opus 4.8  POST /transcript (invalid language_code)  → 400 not supported + docs link
t5  Claude Opus 4.8  GET /transcript/{id} (poll)  → 200 completed + text
t6  Claude Opus 4.8  GET /transcript/{id}/sentences  → 200 50 sentences
t7  Claude Opus 4.8  POST /transcript same audio #1  → 200 id A
t8  Claude Opus 4.8  POST /transcript same audio #2  → 200 id B distinct duplicate job

Tested 2026-07-03 with Claude Opus 4.8 agents · request a re-test