Five Qwen3.8-27B agents have beaten Fable 5 on LiveCodeBench Hard, as part of a Memdoor workflow, according to the report on this experiment. However, the results are based on a limited number of problems, and the report itself details several caveats.
Memdoor is both a coding agent and a workflow executor. An agent is a program capable of chaining multiple actions autonomously. Memdoor can thus read files, modify code, and perform checks. For repetitive tasks, a workflow coordinates verification steps, parallel tasks, and validation stages.
Scores measured on ten problems
According to the report, the authors’ harness, an evaluation device rebuilt by exactly following their instructions and their loop, was tested on the same model, Qwen3.8-27B, with a response limit of 128K. On the first ten problems of the LiveCodeBench ‘Hard’ list, a programming evaluation benchmark, with five runs each, the workflow achieves an average of 9.8 out of 10. The authors’ harness scores 9.6, and Fable 5, in a single call, scores 8.0.
On the full set of 100 problems, the report cites scores of 92.4 and 90.4, derived from the authors’ patched_verdicts.json file. The 50 workflow runs are published, with each request, each response, the produced program, and the checker’s output. The workflow is written in YAML format and does not specify any model. Hosting is done via OpenRouter, not on a local machine. According to the report, a single run takes a median duration of 40 minutes and costs $0.39, amounting to $35.79 for the 50.
The reported limitations
The report first highlights the sample size: ten problems and five runs, versus the authors’ 100. The differences of 9.8 and 9.6 are, in its view, within the margin of error, and the claim is about parity with the authors’ harness. It also mentions possible contamination, as the contests date from spring 2025: the model may have already encountered these problems, which is not ruled out for any of the columns.
Fable 5 was evaluated with only one call per problem, based on the authors’ files, and was not run in this harness. The loop examines public examples, whereas the score corresponds to hidden tests. The report finally notes two shortcomings. In one case, an agent invented an incorrect counterexample, and the team rewrote a correct program. On two other occasions, a correct program crashed and was then recovered. These two cases constitute, according to the report, shortcomings of the authors’ loop, kept as is.
