Artificial intelligence assistants must also be able to answer poorly phrased questions, not just precisely formulated ones. Researchers present XYBench, a benchmark (a standardized test suite for comparing models) of over 8,000 questions, which measures how assistants react when a request is based on a misconception. The work received the Best Resource Paper Award at the EMNLP 2026 conference.
The case described by the authors is the XY problem: a person wants to achieve a goal (X), assumes a given method (Y) will accomplish it, and asks for help on that method. Their example is icing sliding off cupcakes. The person seeks to thicken the cream, while the real problem is that the cakes are still too warm to be iced.
How the benchmark was built
XYBench is based on 8,115 questions drawn from four sources: 1,557 questions from Stack Overflow, 184 from Stack Exchange, 102 manually crafted examples, and 6,272 questions generated from WikiHow guides. For each question, the researchers identified the stated request (Y) and the real goal (X), which can be inferred from the context. For forum questions, the top-rated human answers were used to establish the goal and possible solutions. For WikiHow, the Sonnet 4.5 model generated questions around plausible misconceptions, which were then refined.
Responses are evaluated on three criteria: the presence of solutions for Y and for X, the emphasis placed on one or the other request, and the ability to steer the user away from the erroneous method. The automatic judge is Claude Sonnet 4.5. Two human reviewers examined 500 responses, with average agreement between 97.5% and 100% for the presence of a solution, between 84.6% and 93.3% for identifying the misconception, and between 74.2% and 79.8% for emphasis.
Mixed results
The researchers tested Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, with one question per response. According to the study, all models more often propose a solution to the stated request (Y) than to the real goal (X). On Stack Overflow, solutions for Y appear in 75% to 86% of responses, compared to 59% to 71% for those targeting X. Models primarily emphasize X in 35% to 41% of cases, and Y in 47% to 57% of cases.
In an ‘oracle’ experiment, models are given both Y and X. Claude Sonnet 4.6’s rate of proposing a solution for the real goal then rises from 59% to 67% on Stack Overflow, but its rate of identifying the misconception drops from 47% to 34%. According to the authors, providing the intent helps but does not consistently close the gap with human responses. A second experiment, where six models, including GPT-4.1, must choose between two responses, shows they select the goal-focused response in 71% to 84% of cases. The researchers see a gap between recognizing good advice and producing it.
The benchmark currently covers English queries and mainly procedural problems. Its extension to other languages, cultures, and conversational contexts remains, according to the authors, an open avenue.
