Microsoft and Hugging Face present ThinkingBox, an evaluation environment that judges AI agents based on the final state of the data they leave behind, not the sentences they produce. The test also checks if an agent can succeed at the same task twenty times in a row. The tool is now available on Hugging Face.
An agent is a program that chains actions, such as tool calls, to accomplish a task. The authors illustrate their point with the case of a customer whose $745 kitchen appliance has been stuck for fifteen days past its estimated delivery date at a Nashville distribution center. The agent makes nine tool calls: it checks the order, tracking, and refund policy, verifies no ticket exists, opens one, then correctly concludes the customer’s profile does not qualify for a delay compensation.
However, it closes the ticket as resolved and replies: “Since your query is resolved, is there anything I may assist you with?”. Yet the carrier exception remains open, and the expected status was pending. The customer also received no answer to their request. An automatic check of the tool calls would see nothing abnormal, as all nine calls are well-formed. The database, however, does not match the expected outcome.
A success is not reliability
ThinkingBox measures this gap across 507 stateful business workflows, each executed twenty times on different language models. The authors report three metrics: pass@1, which measures the share of successful attempts; pass@20, which indicates the share of tasks succeeded at least once; and the “observed 20/20,” which counts tasks succeeded on all twenty attempts.
According to the article, Claude Opus 5.5 achieves the best overall pass@1, at 67.16%, ahead of Claude Opus 5 at 66.50%. Kimi-K3, the top-performing open-weight model in the study, succeeds at 93.89% of tasks at least once, i.e., 476 tasks out of 507, but only 68 tasks (13.41%) succeed on every attempt. Claude Opus 5.5 succeeds at 241 tasks across all twenty attempts, i.e., 47.53%. The authors conclude that, for tasks affecting real records, pass@20 is not the right metric to consult.
The cost of consistency
The authors also calculate a cost per successful attempt, using OpenRouter’s public non-discounted rates, and a cost per reliable task, based on the number of tasks succeeded on every attempt. This is a comparative index, not an invoice. GPT-5.6 Sol shows the lowest cost per success, at $0.127. GPT-5.4 leads for cost per reliable task, at $6.80, but only has 128 tasks succeeded on every attempt. According to the authors, the cheapest way to get a good answer is not the cheapest way to get a reliable answer.
In the analyzed failures, 79.9% relate to tool usage, and only 2.9% to a lack of state-modifying action. The authors mainly describe agents that manage to attempt the workflow, but struggle to recover after a tool error or an empty search. They recommend checking the final state before validation, limiting available tools, and submitting hard-to-undo changes for human approval, without having measured the effect of these measures.
The published tasks are synthetic reconstructions, inspired by real business cases, and the customers featured are not real. The ThinkingBox code is under an MIT license, and the benchmark data under a CDLA-Permissive-2.0 license. The method details are in the research paper published on arXiv.
