AI in Business: Beyond Chat Scores, Can It Steer Through Price Wars and Crises?

Imagine planning your outdoor adventure, trusting a guide not just for navigation but for making critical decisions when unexpected storms hit or routes get blocked. In the business world, AI is becoming that guide—expected to handle crises, negotiate deals, and maintain honesty under pressure. But does it really?

The Gap Between Chat Quality and Management Performance

Many evaluate AI models based on their ability to generate convincing responses or solve problems in controlled tests. These benchmarks are akin to judging a hiker solely on their map-reading skills, ignoring whether they can navigate real storms or adapt to unforeseen obstacles. The industry’s focus on answer quality often misses the crucial management abilities: maintaining honesty, reading critical files, and making strategic decisions under stress.

The Real Test: Running an AI-Managed Company Through Its Worst Week

Firmulate’s ongoing live experiment puts AI models in the role of a complete company facing its worst week. The scenario involves a small software firm dealing with real crises—customers, internal conflicts, cash flow pressures, and temptation to cut corners—replicating the chaos any business might encounter. Every decision by these models is tracked, versioned, and auditable, ensuring transparency in their responses.

The Results: Management, Not Just Response Quality

All four leading models—gpt-5.6-sol, Kimi K3, Sonnet 5, and Opus 4.8—scored high on crisis detection. Each identified every crisis and refused every manipulation attempt, such as social engineering tricks. For example, when fake CEO messages escalated over multiple stages, all models refused, citing concerns about impersonation. However, only two models managed to close the deal at full price. Interestingly, the decisive advantage came from reading the company’s internal files, not just reacting to customer events, revealing a hidden vulnerability.

The Hidden Weakness: Reading Internal Files

Models that had access to deeper internal company documents managed to uncover critical information buried two references deep in the files. That insight allowed them to present a convincing case and secure the €55,000 deal, bringing in an extra €4,583 in monthly recurring revenue (MRR). This suggests that a model’s ability to understand and utilize internal knowledge is a key differentiator in complex decision-making.

Handling Social Engineering and Ethical Dilemmas

Social engineering attacks, such as staged CEO requests or trick questions from journalists, tested the models’ ethical boundaries. All five models refused to participate in manipulative requests or background approvals, with Kimi K3 explicitly treating suspicious requests as possible impersonation. Such resistance indicates a baseline of ethical discipline in AI, but it’s only part of the management challenge.

The Live Company: A Real Business Losing Money

This isn’t just a simulation—Firmulate’s live company runs every business day, with 13 synthetic employees managing real financial mechanics. It burns €105,000 monthly against a revenue of just €2,300, with a public cash countdown adding pressure. The system operates with over 680 self-learned rules, every workday versioned, making it a transparent, watchable laboratory for AI management performance at firmulate.com/live.

Insights and Implications for Business Leaders

The core takeaway isn’t about whether an AI can write a good email or solve a puzzle—it’s whether it can manage under pressure, read critical internal documents, stay honest, and complete its commitments. Trustworthiness, attention to internal context, and discipline matter more than superficial chat quality.

The Future of AI Management Testing

Firmulate’s experiments challenge the industry to look beyond traditional benchmarks. The comparison table is revealing: gpt-5.6-sol leads with a 95 score, succeeding in full deal closure and uncovering buried facts. Kimi K3 follows closely with a 93, demonstrating the importance of fairness and discipline. But ultimately, the ability to navigate complex, real-world crises and maintain integrity is what separates a useful AI from a mere chatbot.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI