Vercel
Vercel's REST API is a clean agent citizen. One POST creates a project and returns its id; a bad name is a 400 that teaches the naming rule with a docs link and a 'Learn More' action; an invalid framework is a typed 400; and a chain works - create a project, add an env var by id, read it back, GET the project. Idempotency is right: the project name is a unique key, so the same name twice returns 200 then a 409 'already exists', no duplicate, without any idempotency-key header. (Vercel's terms explicitly permit publishing reproducible benchmarks.)
Cross-validated onClaude Opus 4.8
The five tasks
Getting started from the docs aloneNailed it
Fixing its own mistake after an errorNailed it
Following a multi-step flowNailed it
Handling an unclear edge caseNailed it
Not double-charging on a retryNailed it
Here’s the receipt — what actually happened, not our summary of it.
The project name is a unique natural key. POST /v11/projects {name:'agentcheck-idem'} -> 200 (created); an identical second POST -> 409 {error:{code:'conflict', message:'Project "agentcheck-idem" already exists.'}}. A dropped-response retry cannot create a duplicate project - no idempotency-key header needed.See everything the AI did (5 steps)
t1 Claude Opus 4.8 POST /v11/projects (create) → 200 project object t2 Claude Opus 4.8 POST /v11/projects (invalid name) → 400 invalid_project_name + docs link t3 Claude Opus 4.8 POST project -> POST /env -> GET /env -> GET project → chain ok: env AC_KEY created + read back t4 Claude Opus 4.8 POST /v11/projects (framework:'not-a-framework') → 400 invalid_framework t5 Claude Opus 4.8 POST /v11/projects same name x2 → 200 then 409 conflict (no duplicate)
Tested 2026-07-04 with Claude Opus 4.8 agents · request a re-test