This source only provides a headline — there is no body to show. Use “Read the original” below to read it at the source.
Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks
arXiv:2609.16023v2 Announce Type: replace Abstract: Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale naively, however, is expensive, and rubric-based evaluation has become the dominant scalable alternative. We ask what happens when the rubrics themselves are not airtight, and whether such flaws can be detected and corrected. We apply RIFT, a global rubric failure taxonomy, to three clinical benchmarks (HealthBench Professional, LiveMedBench, and MentalHealthBench), and find failure modes are meaningful: on HealthBench Professional an LLM judge flags 29.6% of criteria as non-atomic and 65.4% as misaligned/rigid, although these rates vary severalfold across five detection judges. Then, we show that these flaws are meaningful and not simply cosmetic. As an example, rewriting bundled criteria of the form "at least one of / all of the following" as equally weighted children and regrading…