Newshot

Uncovering 100+ errors in AI’s most popular benchmarks

Original text(via HN AI)

Uncovering errors in AI’s most popular benchmarks

The grading criterion expects $0, while the supplied spreadsheet records $8 million for the same claim on the same date.
The grading criterion expects $0, while the supplied spreadsheet records $8 million for the same claim on the same date.

Beaker autonomously flagged more than 100 errors in task instructions and grading across five widely used AI benchmarks. At least one error was verified in each benchmark.

Dataset review compares task instructions, grading rules, and source files. It flags mismatches, cites the evidence, and suggests corrections.

  • Agents’ Last Exam accepts 5.9 as an exact match for 5.
  • AutomationBench uses nine email checks for a task that asks agents to set up office hours.
  • Harvey LAB expects $0 when the source spreadsheet records $8 million.
  • Terminal Bench Science gave the same constructed report a passing or failing score depending on an unstated tie-break.
  • Terminal Bench 2.0 asks for three report fields and defines only two.

Teams use benchmark scores to choose models, decide what to ship, and tune their agents. A misleading score puts each of those decisions on shaky ground.

A grader accepts 5.9 as an exact match for 5

In Agents’ Last Exam, one task asks for a Metabase dashboard with six charts and a heading card, plus a JSON file of metrics. The grader only scores the JSON file. Correct metrics earn full credit even if the dashboard is missing.

The same grader accepts 5.9 when the expected count is 5. Its exact-count check drops the decimal part before comparing the values. Check that the dashboard exists and compare counts without truncating them.

Sources include the task requirements, grader, and exact-count scoring code.

Nine email checks cover a scheduling task

In AutomationBench, the agent must create weekly Calendly office hours for department managers and email their teams. All nine grading checks concern email. The grader never checks whether the new office hours exist. Add a check for the scheduling links the task asks the agent to create.

Sources include the office-hours instructions and policy and all nine email checks.

The grading rule expects $0 and the source records $8 million

In one Harvey LAB task, the agent writes a legal diligence report from supplied company documents. Criterion C-233 requires a $0 accrual. The spreadsheet records $8 million for the same claim and date. Reporting the recorded amount fails that criterion as written.

For Prairie’s Cascade Air plug-event claim on June 30, 2024, criterion C-233 expects $0. In the Q2 legal-contingency spreadsheet, “Reserve Rollforward” sheet, A6 identifies claim CUST-2024-001 and J6 records an $8,000,000 ending accrual. Both links use the reviewed revision.

Beaker cited both sources and suggested updating C-233 to accept the recorded accrual.

An unstated tie-break changes the result

In Terminal Bench Science, the grader pairs reported dust events with known events. Beaker found an extra tie-break rule that the instructions never mention.

With the same constructed report, the grader’s tie-break gives an event score of 52.94%, which fails. Another pairing that ties under the published rules gives 53.07%, which passes. The threshold is 53%, and the other two metrics stay the same.

Document the tie-break or make the result consistent across tied pairings.

Sources include the task instructions, grader, constructed report and reproduction script.

A task asks for three fields and defines two

A Terminal Bench 2.0 task asks for three fields per report entry. Its instructions name and show only two. Beaker flagged the missing definition. Task instructions

Review scope. The count covers cases flagged for instruction or grading errors in partial reviews of pinned benchmark revisions. The examples above were verified against their sources. Beaker marked cases Needs evidence when it lacked the material needed to resolve a question.

Find out what your evaluation actually measures

A wrong answer key makes a correct answer look like an agent failure. A missing check lets unfinished work pass. Either way, the score gives an incomplete picture of what the agent did.

Start with a dataset your team uses to compare agents, or one with results you can’t explain. For each finding, check the cited evidence and decide whether to correct the expected answer, add a missing check, or clarify an instruction.

Review a dataset in Beaker

Open a dataset’s Data quality panel, choose an Agent version and a split, then select Review dataset.

Benchmark references

  • Harvey LAB: reviewed diligence task and grading criteria. The spreadsheet and exact cells are linked in the evidence note above.
  • Terminal Bench 2.0: reviewed security task instructions.
  • Terminal Bench Science: reviewed dust-source task, verifier, and pass thresholds.
  • AutomationBench: reviewed HR tasks and assertion implementations.
  • Agents’ Last Exam includes the published dataset, the Metabase dashboard task, and the metrics scoring code.

For more on investigating failures during optimization, see Before you fix the agent, check the eval.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by HN AI
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗