📊 Full opportunity report: The AI Leaderboard That Matters Starts After The Demo Ends on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
A new live benchmark tests AI models in managing a simulated company during crises, emphasizing management decision-making over simple response quality. Results show models excel at diagnosis but struggle with execution and trust, revealing key gaps in current evaluation methods.
Firmulate has launched a live experiment that evaluates AI models’ ability to manage a simulated company during its most challenging week, marking a significant shift from traditional benchmarks focused on chat or code quality. The results, announced in July 2026, reveal that management skills—such as diagnosis, decision-making, and trust—are critical areas where models excel or falter, emphasizing the need for new evaluation standards that go beyond response accuracy. For more details, see the original analysis.
The experiment involved five AI models competing in a scenario where they had to handle crises, negotiate deals, and maintain trust within a small software company experiencing severe financial and operational stress. The models were scored on their ability to identify issues, communicate effectively, and complete critical tasks, with a strict trust cap: even a single breach resulted in disqualification. The top performer, gpt-5.6-sol, scored 95 out of 100, while the lowest, Opus 4.8, scored 73.
Despite all models successfully diagnosing crises and resisting manipulation attempts—such as fake CEO messages—only two signed a €55,000 deal based on their own analysis. The failure to close deals often stemmed from missing critical facts buried in company files, illustrating that models can sound informed but still lack effective retrieval of vital information. The experiment also tested manipulation resistance, with all models refusing to disclose sensitive information when pressured, demonstrating a strong safety posture. This approach is discussed in AI’s Next Benchmark Is the Week After the Demo.
However, the most thorough model, Opus 4.8, which added extensive rules and deep analysis, finished last in execution, revealing that more activity and guidance do not necessarily translate into effective management. The results suggest that current AI benchmarks overvalue activity and superficial effort, neglecting whether the work truly reaches completion through appropriate channels. Learn more in the original analysis. The experiment underscores that management quality—prioritization, trust, and outcome—must become a core dimension of AI evaluation.
Implications for AI Evaluation Standards
This experiment demonstrates that traditional benchmarks—focused on chat responses or code—miss critical aspects of management, such as trustworthiness, decision quality, and the ability to handle complex, real-world scenarios. The findings argue for a shift toward evaluating AI in operational contexts, where the ultimate goal is not just accurate diagnosis but effective, trustworthy management of organizational consequences. For organizations deploying AI agents, this means asking whether models can read relevant files, escalate issues appropriately, and maintain honesty under pressure—skills vital for real-world application and risk management.
AI management decision-making tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limitations of Conventional AI Benchmarks
Most existing AI evaluations rely on static tests, such as coding competitions or chat arenas, which measure technical output or conversational quality. These benchmarks do not assess how models perform when managing ongoing crises, making decisions with incomplete information, or maintaining trust over time. The Firmulate experiment builds on prior work by embedding models in a live, dynamic environment where they must manage a small company’s day-to-day operations under stress, providing a more realistic measure of AI readiness for enterprise use.
Previous efforts have highlighted the gap between technical proficiency and practical management, but few have tested models in such a comprehensive, real-time scenario. The July 2026 Crucible League’s results serve as a benchmark for what AI can and cannot do in organizational management, emphasizing that success depends on more than just generating correct answers—it requires responsible, context-aware decision-making.
“This experiment exposes the management gaps in current AI models, revealing that diagnosis alone is not enough; execution, trust, and context are equally vital.”
— Thorsten Meyer, Lead Researcher at Firmulate
Remaining Questions About Model Performance
While the experiment provides valuable insights, it remains unclear how these models would perform in different industries or with larger, more complex organizations. The impact of training data, fine-tuning, and specific use cases on management effectiveness has yet to be fully understood. Additionally, long-term trust and consistency over extended periods are still untested, raising questions about how models will behave in continuous operational settings.
Next Steps for AI Management Benchmarks
Future work will likely include expanding the scope of live management experiments to diverse industries and larger organizations, testing models over longer periods, and integrating human oversight more deeply. Companies considering AI for operational roles should evaluate models against these new benchmarks, focusing on their ability to read organizational context, escalate issues appropriately, and maintain trust. The ongoing development of such standards aims to ensure AI deployment aligns with real-world management needs and risk mitigation.
Key Questions
Why does management ability matter more than chat quality?
Management ability reflects an AI’s capacity to handle complex, real-world tasks such as decision-making, trust maintenance, and crisis resolution—skills critical for organizational success that chat quality alone cannot measure.
What does the experiment reveal about AI safety and manipulation resistance?
All models successfully refused manipulation attempts, such as fake CEO requests, indicating strong safety postures. However, safety alone does not guarantee effective management, as execution gaps remain.
Can current AI models manage real companies effectively?
While they show promise in diagnosis and safety, models still struggle with completing tasks, retrieving critical facts, and executing decisions reliably—areas needing further development before deployment in operational roles.
What should organizations ask when evaluating AI for management tasks?
Organizations should assess whether models can read relevant files, escalate issues properly, remain honest under pressure, and complete tasks through appropriate channels, not just response quality.
What is the significance of the July 2026 results?
The results serve as a benchmark for future AI management evaluation, emphasizing that success depends on operational skills like trust, decision-making, and context awareness beyond traditional benchmarks.
Source: ThorstenMeyerAI.com
