The agent-usability leaderboard
If my agent can’t use it,
I probably won’t either.
We had AI agents try to actually use well-known AI products. No humans helped. Here’s what happened.
Agent-friendly36
Retried a dropped send three times, and it sent three emails. There's a seatbelt against this - but it's off unless you ask for it.
Wrote the same record three times on purpose, and it kept exactly one. Retrying just can't hurt you here.
Start a crawl without a limit and it heads straight for the 10,000-page default - 2,671 pages in 8 seconds, no prompt, no guardrail - so an agent that forgets the limit burns credits fast.
Fed it a bad parameter value, and the error came back listing every valid option by name. An agent can self-correct without ever opening the docs.
Pointed it at a link that served a web page instead of audio, and the error said exactly that - the remote server sent text, not sound. Every error names the real cause.
Ran code that crashed, and instead of a vague failure it came back with the error's exact name, message, and where it broke - everything an agent needs to fix its own code and retry.
When a page it fetched actually failed, Scrapfly tells you the truth — and every error it throws says exactly what to fix, so an agent can correct itself without opening the docs.
Get a request wrong and Exa practically tells you how to fix it — it lists the allowed values or explains why a combination isn't allowed, so an agent recovers on its own.
Agent-hostile24
No door for an agent to even knock on.
No public API — agents can't get in.
Send the wrong version setting and Cal.com replies 'this address doesn't exist' instead of 'wrong version' - so an agent that trusts the error hunts for a page that was there all along.
Ask it to read a page that's actually broken and it hands back the error page as if everything worked — your agent can't tell, and passes the junk straight downstream.
Ask for too many results and You.com hands back nothing at all — your agent reads it as 'no matches found' when really it just asked for too much.
Call a tool that doesn't exist, or one you haven't connected, and Klavis returns is_error=False — an agent trusting that flag treats the failure as a success. The real error is only in the text.
The response dropped, the agent retried, and the same recording got transcribed and paid for twice. Nothing caught the repeat.
The response dropped, the agent retried, and the same video got encoded and paid for twice. Nothing caught the repeat.
Every verdict is backed by a real transcript — tap to see it.
All verdicts in this batch were produced by Claude-based agents under identical conditions. Different models may perform differently — multi-model testing is on the roadmap.