Migrating a production AI agent to GPT-5.6 | Ploy
Switching Ploy's production agent from Claude Opus to GPT-5.6 revealed that "the model" is actually a stack of provider-specific behaviors—tool-calling conventions, cache architectures, reasoning replay—that your system has invisibly specialized around, requiring surgical fixes at every layer.
Read Original Summary used for search
TLDR
• GPT-5.6 invents plausible values for unused optional tool parameters (offset: 0, timeout: 120000), causing 52-64% of file reads to silently return empty—fix by rewriting schemas to required-but-nullable with anyOf: [T, null]
• Claude caches org-wide with no limits; GPT-5.6 requires explicit per-workspace cache keys with ~15 req/min throughput per key—misconfigured caching made GPT-5.6 look 50% more expensive when it wasn't
• Eval harnesses are tuned to incumbent models: their tool-call budgets fit Opus's sequential style but failed GPT-5.6's correct parallel calls, making ~33% of raw failures trace to harness assumptions, not model quality
• GPT-5.6 writes dramatically leaner code (2.5K chars vs 17K for comparable output) and finishes builds in half the time at 27% lower cost, but converges toward uniform layouts without steering
• Model migration is infrastructure migration: "universal" SDKs hide that tool schemas, caching, and reasoning replay are provider-specific contracts your stack has quietly specialized around
In Detail
Ploy migrated their production marketing-website-building agent from Claude Opus 4.8 to GPT-5.6 Sol, the first model in four months to beat Opus on their evals. The migration delivered 2.2× faster builds at 27% lower cost, but required discovering that "the model" is actually a stack of provider-specific behaviors their system had invisibly adapted to. Despite using Vercel's AI SDK—a universal LLM abstraction—switching models meant fixing tool schemas, caching architecture, and reasoning replay one eval failure at a time.
The first surprise: their eval harness was tuned to Opus. Tool-call budgets sized for Opus's sequential style failed GPT-5.6's parallel calls on cases it was solving correctly. The executor didn't support batched file reads, which Opus rarely used but GPT-5.6 uses constantly. Roughly a third of raw failures traced to harness assumptions, not model behavior. The second surprise: GPT-5.6 sends all 25 tool parameters every time, inventing plausible values for unused ones (offset: 0, siteId: "00000000-..."). These looked real, causing 52-64% of file reads to return empty. Prompting didn't fix it; they rewrote optional params as required-but-nullable using anyOf: [T, null], stripping nulls at the provider boundary. Empty reads dropped to 0%, tool calls dropped 30%.
The most instructive difference was prompt caching. Claude caches org-wide with cache_control breakpoints and no throughput limits, hitting 92-96% cache rates. GPT-5.6 dropped partial-prefix matching and requires explicit prompt_cache_key markers—and the key is part of cache identity. Per-conversation keys meant 0% first-call hits; one global key exceeded the ~15 req/min per-key budget and spilled to cold nodes. Per-workspace keys were the sweet spot, but cross-workspace sharing of the static 29K-token prefix became structurally impossible. Before the fix, GPT-5.6 looked 50% more expensive; after, first-call hits went from 0% to 83.7% and total cost landed below Opus. Every dollar of the gap was cache misconfiguration, not model pricing. GPT-5.6 also replays prior-turn reasoning as server-side references by default, causing intermittent "Item not found" errors; store: false makes replay self-contained. The result: GPT-5.6 writes dramatically leaner code (2.5K chars vs 17K for comparable pages), finishes builds in half the time, but tends toward clean, uniform layouts unless steered.