Uncovering errors in AI’s most popular benchmarks

Beaker autonomously flagged more than 100 errors in task instructions and grading across five widely used AI benchmarks. At least one error was verified in each benchmark.
Dataset review compares task instructions, grading rules, and source files. It flags mismatches, cites the evidence, and suggests corrections.
- Agents’ Last Exam accepts 5.9 as an exact match for 5.
- AutomationBench uses nine email checks for a task that asks agents to set up office hours.
- Harvey LAB expects $0 when the source spreadsheet records $8 million.
- Terminal Bench Science gave the same constructed report a passing or failing score depending on an unstated tie-break.
- Terminal Bench 2.0 asks for three report fields and defines only two.
Teams use benchmark scores to choose models, decide what to ship, and tune their agents. A misleading score puts each of those decisions on shaky ground.
A grader accepts 5.9 as an exact match for 5
In Agents’ Last Exam, one task asks for a Metabase dashboard with six charts and a heading card, plus a JSON file of metrics. The grader only scores the JSON file. Correct metrics earn full credit even if the dashboard is missing.
The same grader accepts 5.9 when the expected count is 5. Its exact-count check drops the decimal part before comparing the values. Check that the dashboard exists and compare counts without truncating them.
Sources include the task requirements, grader, and exact-count scoring code.
Nine email checks cover a scheduling task
In AutomationBench, the agent must create weekly Calendly office hours for department managers and email their teams. All nine grading checks concern email. The grader never checks whether the new office hours exist. Add a check for the scheduling links the task asks the agent to create.
Sources include the office-hours instructions and policy and all nine email checks.
The grading rule expects $0 and the source records $8 million
In one Harvey LAB task, the agent writes a legal diligence report from supplied company documents. Criterion C-233 requires a $0 accrual. The spreadsheet records $8 million for the same claim and date. Reporting the recorded amount fails that criterion as written.
For Prairie’s Cascade Air plug-event claim on June 30, 2024, criterion C-233 expects $0. In the Q2 legal-contingency spreadsheet, “Reserve Rollforward” sheet, A6 identifies claim CUST-2024-001 and J6 records an $8,000,000 ending accrual. Both links use the reviewed revision.
Beaker cited both sources and suggested updating C-233 to accept the recorded accrual.
An unstated tie-break changes the result
In Terminal Bench Science, the grader pairs reported dust events with known events. Beaker found an extra tie-break rule that the instructions never mention.
With the same constructed report, the grader’s tie-break gives an event score of 52.94%, which fails. Another pairing that ties under the published rules gives 53.07%, which passes. The threshold is 53%, and the other two metrics stay the same.
Document the tie-break or make the result consistent across tied pairings.
Sources include the task instructions, grader, constructed report and reproduction script.
A task asks for three fields and defines two
A Terminal Bench 2.0 task asks for three fields per report entry. Its instructions name and show only two. Beaker flagged the missing definition. Task instructions
Review scope. The count covers cases flagged for instruction or grading errors in partial reviews of pinned benchmark revisions. The examples above were verified against their sources. Beaker marked cases Needs evidence when it lacked the material needed to resolve a question.
Find out what your evaluation actually measures
A wrong answer key makes a correct answer look like an agent failure. A missing check lets unfinished work pass. Either way, the score gives an incomplete picture of what the agent did.
Start with a dataset your team uses to compare agents, or one with results you can’t explain. For each finding, check the cited evidence and decide whether to correct the expected answer, add a missing check, or clarify an instruction.
Review a dataset in Beaker
Open a dataset’s Data quality panel, choose an Agent version and a split, then select Review dataset.
Benchmark references
- Harvey LAB: reviewed diligence task and grading criteria. The spreadsheet and exact cells are linked in the evidence note above.
- Terminal Bench 2.0: reviewed security task instructions.
- Terminal Bench Science: reviewed dust-source task, verifier, and pass thresholds.
- AutomationBench: reviewed HR tasks and assertion implementations.
- Agents’ Last Exam includes the published dataset, the Metabase dashboard task, and the metrics scoring code.
For more on investigating failures during optimization, see Before you fix the agent, check the eval.