When an enterprise AI system delivers an unreliable answer, the usual diagnosis points to the model. However, according to the source article, a more frequent and harder-to-spot failure lies elsewhere: a large language model (LLM) can reason correctly from a poorly selected context.
The most common remedies are a more powerful model, a longer prompt, more retrieved documents, or a larger context window, i.e., the amount of information a model can process at once. Each can help, but none directly solves this problem. In analytical systems, the calculation can be correct and the explanation coherent, and the answer can still be wrong: the system chose the wrong evidence, followed the wrong link between tables, or gave an ambiguous business term a definition the user did not have in mind. The error occurs before any computation.
Take a simple question: how many hours does Cam work per day on average? The word “day” is undefined. It could mean each calendar day in the period, a standardized work year, each schedule record, or each date Cam actually worked. Dividing total hours by calendar days yields a lower average than dividing by days actually worked as soon as Cam has days off. A classic control, which checks that totals match, does not necessarily detect this wrong choice of denominator.
A business question, an analysis program
The question “Which brand improved trust this quarter?” looks like a simple search. Answering it, however, requires deciding several points: which survey question measures trust, which survey waves constitute “this quarter”, what is the respondent scope, whether results are weighted, and what counts as an improvement.
The article takes an illustrative case. Two pivot tables seem to match the request, with descriptions differing by only a few words. The first, weighted, covers all respondents and shows a slight increase. The second, unweighted, covers only respondents familiar with the brand and shows a slight decrease. A similarity-based search tool would rank them almost equally. A system that picks the first result provides a sourced and reasoned answer, which can be wrong, without the choice made appearing in the final output. A system based on a journey, however, notes the difference in population and weighting and asks the user which scope to use. The cost of this clarification question is, according to the author, far lower than that of a false figure that looks right.
Benchmarks that measure systems
Research points in the same direction. Spider 2.0, an enterprise text-to-SQL benchmark comprising 632 problems from real workflows, was published in late 2024. Its top-performing system, an agent built on o1-preview, solved about 17% of the tasks, compared to about 91% for the best systems on the original Spider benchmark. The ranking has since evolved: its Snow track shows a score of 96.7%, submitted by a vendor, from an agent system. Scores are submitted by the teams themselves, and the article notes that the best results rely on systems designed around the model rather than on models alone.
Lengthening the context is not enough either. A Chroma study titled “Context Rot” finds that models become less reliable as input length increases, even on simple tasks, and that similar-looking distractors worsen the problem.
A journey rather than a heap
Faced with uncertainty, the reflex is to give the model more information. Yet this can multiply competing interpretations. The described approach involves organizing the context sequentially: first the study intent, then the questionnaire structure, the questions and their definitions, the segments, the markets, the waves and weighting rules, and finally the pivot table containing the evidence. Each layer inherits the constraints of the previous one.
Anthropic presents context engineering as the natural next step from prompt engineering, defining it as managing an agent’s entire context, treated as a finite resource. The article considers this framing relevant but places it a step earlier: it’s about understanding what the question means and which pieces of evidence can answer it. For the brand trust case, the process follows five steps: resolve intent, apply study rules, identify sources, validate the journey, then compute with an authorized tool and present the result with its definition, respondent base, weighting, and source.
The author finally emphasizes traceability. Every result must be traceable back to its source table, its definition, its filters, its weighting, its period, and the calculation used.
