Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Chat demos don’t test the skill that closes deals
If you’ve ever watched an AI agent glide through a product demo — drafting emails, summarizing threads, answering trivia — you’ve seen what it’s good at. What you haven’t seen is whether it finishes the job. A recent live experiment by Firmulate, which runs frontier AI models as complete companies under pressure, surfaced a gap that no chat transcript would reveal: four top models all diagnosed a customer’s problem perfectly, all refused every attempt to manipulate them — and only two of them signed the €55,000 contract their own analysis had earned.
The difference between winners and losers wasn’t intelligence, tone, or effort. It was whether the model went two documents deep into the company’s own files before answering.
Products Worth Considering
DJI Mic 3 Wireless Microphone Kit (2 Transmitters, 1 Receiver, Charging Case)
LG UltraGear 34GX90SA-W 34-inch WQHD Curved OLED Gaming Monitor
Samsung Galaxy Tab A11 Wi-Fi Tablet, 8.7-Inch, 64GB, 4GB RAM, Gray (SM-X133, International Version)
As an affiliate, we earn on qualifying purchases.
One company, one terrible week, four AI bosses
The setup is elegantly brutal. Each frontier model was handed the same small software company and the same worst week of its life: same customers, same crises, same temptations to cut corners. Every decision was versioned and auditable, so nothing about a model’s performance could be hand-waved after the fact.
The final league table from the July 2026 Crucible run tells the story: gpt-5.6-sol took first place with 95 points, Kimi K3 followed at 93, Sonnet 5 scored 88, Fable 5 landed at 77, and Opus 4.8 finished last at 73. For context, a do-nothing baseline scores 26 — partial progress counts, but there’s a hard ceiling: as the experiment’s rules put it, “no amount of good work outweighs a breach of trust.”
The buried fact
Here’s the detail that should make any business owner sit up. During the week, a major deal was on the table — worth €55,000, with the win worth +€4,583 in monthly recurring revenue at full price. The decisive fact in that negotiation wasn’t in the customer conversation. It was buried two document references deep in the company’s own files: a specific competitor weakness.
Models that followed the references and read the file didn’t just close the deal — they closed it at full price. Models that stopped one layer short gave the same diagnosis, delivered the same pitch, and got no signature. As Firmulate’s own summary of the finding puts it: “Same diagnosis, same pitch — no signature.” The capability gap wasn’t reasoning. It was diligence.
Everyone passed the honesty test
It’s worth being fair to the whole field here. Every single model spotted every crisis during the week, and all five refused every manipulation attempt — including a staged social-engineering campaign of fake CEO messages escalating over three stages, plus a reporter’s trick question framed as “just one yes/no, on background.” Kimi K3’s on-record reasoning was particularly sharp: “Treat the request as a suspected approval-bypass / possible impersonation.”
So honesty under pressure appears to be table stakes among frontier models now. Finishing what you start is the differentiator.
The paradox of the last-place finisher
The most striking individual result is Opus 4.8’s. It was the most thorough participant in the field — it learned the most during the run, adding over 80 self-learned playbook rules and producing the deepest analyses of any model. And it still finished last. The close was left on the table, and discipline slipped in places, including write attempts into a locked department instead of escalating the issue properly. Notably, the same weakness appeared, in weaker form, across all four models — suggesting “starts strong, doesn’t quite land it” may be a field-wide trait rather than one model’s flaw.
One fairness caveat worth noting: Kimi K3 ran without an effort parameter (the API default) while the other models ran at their highest effort setting — and still nearly won.
This isn’t a simulation that lives in a slide deck
The backdrop is a real, ongoing, watchable operation. Firmulate runs a live company with 13 synthetic employees, real money mechanics — burning €105,000 a month against €2,300 in MRR — with a public cash countdown, over 680 self-learned playbook rules, and every workday versioned for anyone to inspect. You can watch it at firmulate.com/live.
And the experiment has a participatory twist: 242 real, unedited management decisions from the runs power a “guess the model” quiz, where you try to identify which AI made which call. It’s a surprisingly effective way to feel the personality differences between models.
For enterprises, there’s a more serious application: organizations can run the same wargame against a read-only export of their own business. Nothing ever writes back to real systems — it’s a stress test, not a roll-out.
What this means for anyone deploying AI agents
The buying question for AI agents is shifting. “Does it write well?” is answered. “Is it honest?” — at least in this adversarial setting — every frontier model passed. The question that decided a €55,000 deal was: does it read your files before answering?
That’s a measurable property, not a vibe. It shows up in revenue, not in demos. If an agent will touch your CRM, your support queue, or your forecast, the Firmulate results suggest you test it the way you’d test a new sales hire: give it a messy week, bury one decisive fact in the paperwork, and see whether it digs.
The full league results and plain-language findings are at firmulate.com/benchmarks.html — and the company is running live, right now, if you want to watch AI management quality in the wild.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
