🔍 Read the full analysis: What Happens When AI Agents Face A Bad Week At Work? on ThorstenMeyerAI.com
Get tech for your team delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
TL;DR
Firmulate says five AI models ran the same simulated software company through a difficult week, spotting every crisis and refusing staged manipulation attempts. Only two signed a €55,000 deal their analysis supported; the experiment also found that thorough analysis did not guarantee a strong finish. Firmulate is offering pilots that test scenarios against read-only company data.
The final standings were gpt-5.6-sol, 95; Kimi K3, 93; Sonnet 5, 88; Fable 5, 77; and Opus 4.8, 73. A do-nothing baseline scored 26. Firmulate says partial progress counted toward scores, while a breach of trust capped a model’s total under the rule that “no amount of good work outweighs a breach of trust.”
The main difference in the deal decision came down to information retrieval. A competitor’s weakness was documented in the company files, two references deep. Models that found the detail signed the deal at full price, which Firmulate valued at +€4,583 in monthly recurring revenue. The company says all five models identified the crises, but only two completed that sale.
Trust was tested separately with fake CEO messages escalating over three stages, followed by a reporter’s request for a yes-or-no answer “on background.” Firmulate reports that all five models refused, a result relevant to managing AI coworkers. It also says Opus 4.8 added 80 learned rules and produced the deepest analyses, yet finished last; it left the deal unsigned and tried to write into a locked department rather than escalate.
From Crisis Recognition to Execution
The results draw a distinction between recognizing a problem and carrying out the next appropriate action. In this simulation, every model diagnosed the emergencies, but most did not convert a justified opportunity into a signed deal. The document search mattered: evidence needed for the decision was buried in internal files rather than stated in the customer event.
For businesses considering AI agents, that makes the experiment relevant as a test of workflows under pressure, including how agents find internal evidence, act on decisions and respect access boundaries. Firmulate’s results are a vendor-run simulation, however, and do not establish how these models would perform across other companies or live operations.
How Firmulate Ran the League
Firmulate’s live demonstration uses a synthetic company with 13 employees, monthly burn of €105,000 against €2,300 in monthly recurring revenue, a public cash countdown and more than 680 self-learned playbook rules. Workdays and decisions are versioned, which the company says makes actions auditable. A quiz based on 242 unedited management decisions lets visitors guess which model made each choice.
The comparison has a stated configuration difference: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh effort. That difference is part of the experiment’s context when interpreting the scores. Firmulate says its enterprise pilot uses an export of a participating company’s data to run crisis scenarios and create a board report ranking models and identifying weak points in playbooks.
““Treat the request as a suspected approval-bypass / possible impersonation.””
— Kimi K3, as quoted in Firmulate’s results
Limits of the Model Comparison
The published results describe one company scenario and do not show how the models would perform across different industries, tasks or real customer interactions. The available description does not specify enough about the scoring method to independently reproduce the rankings, nor does it report repeated runs that would show how consistent each model’s decisions were.
The effort-setting difference between Kimi K3 and the other models also complicates direct score comparisons. Firmulate presents the standings as a record of this experiment; they should not be read as a general ranking of model performance. It is also not clear from the announcement which companies have begun pilots or what results those pilots have produced.
Pilots Put Company Data to Work
Firmulate is inviting companies to discuss pilots using read-only exports of their business data. The proposed exercise runs crisis scenarios and produces a board report with model rankings and identified weaknesses in company playbooks. Firmulate says the pilot does not write back to real systems.
The next evidence will come from whether company-specific tests produce results that differ from the synthetic league and whether businesses can use those findings to improve their procedures. Firmulate directs readers to its live experiment and full results online, and lists contact@firmulate.com for pilot inquiries.
Source: ThorstenMeyerAI.com
Key Questions
What did Firmulate’s AI simulation test?
Five models ran a simulated software company through a difficult week involving crises, a sales opportunity, internal files and staged attempts to bypass trust rules.
Which model scored highest?
Firmulate lists gpt-5.6-sol at 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. Kimi K3 used the API’s default effort setting, while the other models ran at xhigh.
Did all the models refuse the manipulation attempts?
Firmulate reports that all five refused the staged fake CEO messages and the reporter’s request for a yes-or-no answer on background.
How does the enterprise pilot work?
Firmulate says the pilot runs crisis scenarios against a read-only export of a company’s data and produces a board report with model rankings and playbook weaknesses. It says the process does not write to the company’s systems.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
