Statistics > Methodology
Title:Variance-adaptive confidence regions for fixed-benchmark binomial mixtures, with application to LLM evaluation
Abstract:We construct confidence intervals for linear functionals $\theta(F_m)=\int h\,dF_m$ of the empirical law $F_m$ of success probabilities $p_1,\dots,p_m$, from independent counts $c_i\sim\mathrm{Bin}(n,p_i)$, as for a language model run $n$ times per problem. Coverage is at least $1-\alpha$ for every $n$, $m$ and fixed $(p_i)$, simultaneously for all bounded $h$. The variance-adaptive (VA) region tests each candidate mixing law with the conditional variance of the tail counts it implies. This variance is linear in the law, so the region is a linear program, and cell envelopes keep it valid off-grid. Regions calibrated for iid sampling target a super-population law and are conservative for $\theta(F_m)$. With all $n$ tail counts, fixed $n$, continuous $h$ and a grid refined as $m\to\infty$, the identification part of VA's width is first-order worst-case optimal, as for any localization with the same index set. Under a rule fixed before the main comparator runs, VA was narrower than a two-stage plug-in band at every $n$ (median width ratio 0.60 at $n=128$), but not than the asymptotic, pointwise AMARI interval at $n=128$; on real rollout pools it was wider than both pointwise and Bonferroni-adjusted AMARI at $n\le128$. For $P(p>0)$ and pass@$k$ with $k\gg n$ the data support only lower bounds.
| Comments: | 27 pages main text plus 57-page supplement. Code: this https URL |
|---|---|
| Subjects: | Methodology (stat.ME) ; Machine Learning (stat.ML) |
| MSC classes: | 62G15 (Primary), 62C12, 60E15 (Secondary) |
| Cite as: | arXiv:2611.21266 [stat.ME] |
| (or arXiv:2611.21266v2 [stat.ME] for this version) | |
| https://doi.org/10.48550/arXiv.2611.21266 arXiv-issued DOI via DataCite |
Submission history
Access Paper:
- HTML (experimental)
Current browse context:
References & Citations
- Google Scholar
- Semantic Scholar
BibTeX formatted citation
Bookmark
Bibliographic and Citation Tools
Code, Data and Media Associated with this Article
Demos
Recommenders and Search Tools
arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.