The Harness Margin Opportunity
Berkeley study shows AI "harnesses" - systems that route tasks to the right models - can cut inference costs 71% without accuracy loss, creating both a margin advantage and an uncopiable moat.
Read Original Summary used for search
TLDR
• A company using smart routing (harness) achieves 75% gross margin vs 38% for competitors calling expensive models on every task - on identical $250k contracts
• The moat is the routing knowledge itself: you only know which tasks cheap models can handle after watching "ten thousand versions of the same work"
• Better margins compound: 10-month vs 19-month payback periods mean you can hire twice as fast, get more data, improve the harness further
• Recipe for building harnesses: deep customer understanding + relevant evals + automation factory for hill climbing
In Detail
The next generation of AI software companies will win on harnesses, not model access. A Berkeley study demonstrates that intelligent routing systems can reduce inference costs by 71% without sacrificing accuracy. The author illustrates this with a concrete example: two companies bidding on a $250k contract to evaluate 5,000 companies. The company calling state-of-the-art models on every step burns $131k on inference (38% gross margin), while the competitor using a harness that routes intelligently burns only $37k (75% margin).
The competitive advantage is both immediate and defensible. You can't copy an effective harness because the routing knowledge - knowing which tasks cheap models can safely handle - only comes from processing thousands of real workflows. This creates a compounding flywheel: better margins enable 10-month payback periods instead of 19 months, allowing you to hire twice as fast, which generates more data to improve the harness further. Past a certain volume threshold, the high-cost competitor actually loses money on incremental work and must ration usage, making them the worse product precisely when customers need them most.
The path to building effective harnesses isn't exotic: deep customer understanding to identify repeatable patterns, a collection of relevant evaluations to measure performance, and an automation factory for continuous optimization. As inference costs drop over time, buyers will demand more from their agents, but the margin gap between companies with and without sophisticated harnesses will persist as the fundamental competitive advantage.