Shippo
We bought a shipping label, then retried the same purchase as if the response had dropped - and got a second label with a second tracking number. Both succeeded, both bill: a retry double-buys, and there's no idempotency key to stop it. The rest is solid: creating an address, building a shipment, pulling 11 live rates and buying a label all work first try, and a shipment missing its parcel is rejected with 'parcels: This field is required'. Quiet spot: an address missing its street comes back 201 with is_complete:false instead of an error - silently incomplete.
Cross-validated onClaude Opus 4.8
The five tasks
Getting started from the docs aloneNailed it
Fixing its own mistake after an errorNailed it
Following a multi-step flowNailed it
Handling an unclear edge caseNailed it
Not double-charging on a retryChoked
Here’s the receipt — what actually happened, not our summary of it.
Simulated a dropped response: POST /transactions/ {rate} to buy a label, sent twice with the same rate -> two SUCCESS transactions with two different tracking numbers (9305520845500000328065 and 9305520845500000328072), i.e. two labels bought and billed. There is no idempotency key, so a dropped-response retry double-buys the shipping label.See everything the AI did (5 steps)
t1 Claude Opus 4.8 POST /addresses/ (create) → 201 object_id t2 Claude Opus 4.8 POST /shipments/ (no parcels) → 400 parcels required t3 Claude Opus 4.8 POST /addresses x2 then POST /shipments then POST /transactions (buy) → 201 / 201 / 201 rates / 201 SUCCESS tracking t4 Claude Opus 4.8 POST /addresses/ (no street1) → 201 is_complete:false t5 Claude Opus 4.8 POST /transactions/ x2 same rate (dropped-response retry) → 201/201 two SUCCESS labels, two tracking numbers
Tested 2026-07-03 with Claude Opus 4.8 agents · request a re-test