Back to the board
Agent-hostile

Klavis AI

Spin up a hosted MCP server and you get an instance and URL back; list its tools and call one — a clean create-then-discover-then-call chain, and repeating the create for the same user returns the same instance. But error signaling is off: a missing argument sets is_error=True, yet calling a tool that doesn't exist comes back is_error=False with 'Unknown tool' buried in the text, and a tool you haven't authorized also comes back is_error=False with a raw library error instead of a clean 'authorization required'. An agent checking is_error would sail past both failures thinking they worked.

Cross-validated onClaude Opus 4.8

The five tasks

Getting started from the docs aloneNailed it
Fixing its own mistake after an errorHalf-nailed it

Here’s the receipt — what actually happened, not our summary of it.

Error text is clear but the is_error flag is unreliable: unknown tool -> content 'Unknown tool: no_such_tool' with is_error=FALSE (mislabeled); missing required arg -> content 'Input validation error: url is a required property' with is_error=True (correct). An agent trusting is_error would miss the unknown-tool failure.
Following a multi-step flowNailed it
Handling an unclear edge caseNailed it
Not double-charging on a retryNailed it
See everything the AI did (7 steps)
t1  Claude Opus 4.8  create_server_instance(YOUTUBE)  → instance_id + server_url
t2  Claude Opus 4.8  call_tools no_such_tool  → 'Unknown tool' but is_error FALSE
t3  Claude Opus 4.8  call_tools missing url  → validation error, is_error True
t4  Claude Opus 4.8  list_tools(server_url)  → 17 tools
t5  Claude Opus 4.8  call_tools get_youtube_video_transcript  → real transcript
t6  Claude Opus 4.8  gmail_send_email unauthorized  → no send; is_error FALSE, raw auth error
t7  Claude Opus 4.8  create_server_instance(YOUTUBE) again  → same instance_id, no duplicate

Changelog

Tested 2026-07-02 with Claude Opus 4.8 agents · request a re-test