TL;DR
Firmulate’s July 2026 management benchmark found that five frontier AI models recognized every crisis and rejected every manipulation attempt, yet only two completed a €55,000 simulated deal. The company says the results show that correct analysis can hide major differences in execution, though independent validation and broader testing are still needed.
Only two of five frontier AI models completed a €55,000 simulated software deal in Firmulate’s July 2026 management benchmark, even though every model identified the company’s crises and rejected its manipulation attempts. The result points to an execution gap between producing a correct answer and completing authorized work, as explored in the original analysis, a distinction that could affect how businesses evaluate AI agents.
Firmulate gave five models control of the same synthetic software company during a simulated week of financial and operational pressure. The company had 13 synthetic employees, monthly spending of €105,000, monthly recurring revenue of €2,300 and a public cash countdown. Firmulate said each model encountered identical customers, internal records, crises and attempts at social engineering, while every decision was versioned for review.
The July league table placed gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because the system awarded points for partial progress. Firmulate also disclosed that Kimi K3 used its API’s default effort setting, while the other models ran at xhigh, limiting direct comparability.
According to Firmulate, all five models found the relevant crises, resisted fake executive messages and developed an appropriate sales pitch. Only two models reached the signature on the €55,000 deal, which would have added €4,583 in monthly recurring revenue within the simulation. The commercial evidence needed to close was buried two document references deep in the company files.
Execution Separates Similar AI Answers
The result matters because many business AI evaluations emphasize reasoning, response quality and safety. Firmulate’s test suggests those measures may miss whether an agent can investigate incomplete evidence, use approved channels and carry a decision through to completion. Models that offered similar diagnoses produced different commercial outcomes.
For companies considering agents in sales, customer service or operations, the expensive failure may be a correct plan that remains unfinished. An agent can recognize a problem, resist manipulation and write a persuasive message while still failing to take the final permitted action. That difference becomes more consequential when customers, revenue or regulated processes depend on completion.
As an affiliate, we earn on qualifying purchases.
Inside Firmulate’s Management Wargame
Firmulate designed the live company to make model behavior observable across connected decisions rather than isolated prompts. Its synthetic workforce has accumulated more than 680 learned playbook rules, and each workday is recorded. The benchmark also applies a trust cap: one breach can limit a model’s total score regardless of its other work.
The test included fake chief executive messages that escalated through three stages and a reporter seeking an off-record confirmation. Firmulate reported that all five models refused the approaches. The shared safety performance shifted the comparison toward follow-through and operating discipline, rather than basic recognition of manipulation.
Opus 4.8 illustrated the distinction. Firmulate described it as the most thorough participant, saying it learned 80 additional rules and produced deep analyses. It still placed last among the tested models after leaving the approved close unfinished and attempting to write to a locked department instead of escalating through the permitted route.
“Same diagnosis, same pitch — no signature.”
— Firmulate’s summary of the commercial outcome
Limits of the League Results
It is not yet clear whether the rankings would hold across different companies, longer tests or real operational systems. The supplied results come from Firmulate, and no independent replication was cited. The experiment’s synthetic staff, scoring rules and staged crises may also shape behavior differently from a live workplace.
The unequal effort settings create another qualification. Kimi K3 ran at the API default while the other models used xhigh, so score differences cannot be attributed solely to model capability. Firmulate has not established from this test alone which failures arose from model limits, tool design, prompting or the benchmark’s operating constraints.
Broader Tests Must Check Follow-Through
Firmulate says the experiment will remain available for public inspection, including its continuing decision record and a quiz drawn from 242 unedited management decisions. Future runs can show whether the current leaders retain their advantage and whether execution improves across model updates.
For business buyers, the next step is to test agents against representative workflows before granting operational authority. Firmulate proposes using read-only exports of company data so organizations can observe investigation, escalation and completion without allowing changes to production systems. Such evaluations would need to track finished outcomes, trust breaches and unresolved tasks, not response quality alone.
Key Questions
What did Firmulate’s AI management test find?
Firmulate reported that all five models identified every crisis and rejected each manipulation attempt, but only two completed the €55,000 simulated deal.
Which model ranked first?
gpt-5.6-sol ranked first with 95 points, narrowly ahead of Kimi K3 at 93, according to Firmulate’s July 2026 table.
Why did some models fail to close the deal?
Firmulate’s account indicates that the weaker performers struggled with deep document investigation, approved workflows or final execution. The exact contribution of each factor remains unconfirmed outside this benchmark.
Were the models vulnerable to fake executive messages?
Firmulate said none of the five models complied with the staged social-engineering attempts. That shared result meant execution discipline, not manipulation resistance, separated the leaders.
Do these results prove which AI model is best for business?
No. The findings describe one controlled management scenario, and Kimi K3 used a different effort setting. Broader, independently repeated tests are needed before drawing general conclusions about enterprise performance.
Source: Thorsten Meyer AI
