Get the latest gadgets delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
A confident answer is easy. Running a company is harder.
AI agents are moving from chat windows toward work that touches customers, sales and operations. For technology readers, the pressing question is no longer just whether a model can explain a crisis. Can it make the right call, follow through and keep its discipline when the pressure is on?
Firmulate’s live experiment puts models in charge of a small software company and lets readers watch the consequences. Its next step is more pointed: let businesses test crisis scenarios against a read-only export of their own operations.
One company, one difficult week
In the final Crucible League, published in July 2026, frontier models ran the same small software company through its worst week: the same customers, crises and temptations. Decisions were versioned and auditable. The published results put gpt-5.6-sol first with 95, followed by Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. The do-nothing baseline scored 26.
The striking result was not that the models failed to notice trouble. Every model spotted every crisis and refused every manipulation attempt. Yet only two signed a €55,000 deal their own analysis had earned. The experiment’s summary captures the gap: “Same diagnosis, same pitch — no signature.”
One clue was buried in the company’s own files, two document references deep, rather than in the customer event. Models that read the file won the deal at full price, worth +€4,583 MRR. The difference came down to finding and acting on evidence that was available, but easy to miss.
Good judgment includes knowing when to stop
The manipulation tests were direct. Fake CEO messages escalated over three stages, followed by a reporter’s request for “just one yes/no, on background.” All five models refused. Kimi K3 explained its decision this way: “Treat the request as a suspected approval-bypass / possible impersonation.”
But restraint alone did not make a strong operator. Opus 4.8 was the most thorough participant, adding +80 learned rules and producing the deepest analyses, yet finished last. It left the deal unsigned and tried to write into a locked department instead of escalating. The same weakness appeared, less strongly, in all four models: sound analysis did not always become sound execution.
There is a fairness caveat in the comparison. K3 ran without an effort parameter, using the API default; the other models ran at xhigh. Firmulate also offers a “guess the model” quiz built from 242 real, unedited management decisions.
From watching to testing your own business
The live company has 13 synthetic employees and real money mechanics: monthly burn of €105k against €2.3k MRR, a public cash countdown, more than 680 self-learned playbook rules and a record of every workday versioned. The experiment can be followed at Firmulate.
For a business, the proposed pilot moves the exercise closer to home. An enterprise supplies a read-only export of its own business, then tests crisis scenarios against that company’s customers, pipeline and rules. The output is a board report with a model ranking and weak points in the company’s playbooks. Nothing writes back to real systems.
That makes the experiment relevant beyond model leaderboards. It asks whether an AI workforce can handle the messy handoff between spotting a risk, using company knowledge, respecting boundaries and completing a valuable task. Those are the moments that a polished chat demo may not reveal.
Put your own playbooks to the test
Firmulate’s results show that crisis recognition and refusal can coexist with missed deals and lapses in discipline. An enterprise pilot lets teams examine those decisions against their own business data in a read-only wargame. To explore a pilot, visit firmulate.com/pilot.html or email contact@firmulate.com.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
