Newshot

Necessary, Not Sufficient, and No Longer Available: Grader Non-Determinism After the Withdrawal of Sampling Controls

Summary

arXiv:2606.26185v3 Announce Type: replace Abstract: LLM-as-judge ("grader") components are now standard in evaluation harnesses, including safety evaluations where a pass/fail verdict may gate downstream deployment decisions. Earlier work, my own included, treated temperature=0 as the remedy for grader non-determinism and asked whether it was sufficient. It was necessary and not sufficient. It is now also unavailable. I report a two-vendor measurement taken in October and November 2026. Anthropic's current models reject the setting outright, and under the API version dated 2026-10-21 all fifteen models listed for my key reject it, the older generations included. On the chat completions endpoint OpenAI's current models accept the parameter and refuse every value but one, so a harness grading through that endpoint is obliged to sample its judge at temperature 1. On the Responses endpoint a request carrying temperature=0 is accepted by four of five current models once reasoning is switche…

This source only provides a headline — there is no body to show. Use “Read the original” below to read it at the source.

Published by arXiv cs.LG
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗