evaluatellm: Statistical Inference for Language Model Evaluations
Treats language model evaluations as statistical experiments and
supplies the inference they require. Provides central limit theorem and
cluster-robust standard errors for evaluation scores, paired and unpaired
model comparisons, variance decomposition when several responses are drawn
per question, control-variate variance reduction, multiplicity adjustment
across benchmark suites, and power and minimum detectable effect
calculations for planning evaluations, following Miller (2024)
<doi:10.48550/arXiv.2411.00640>. For evaluations scored by a model judge,
implements agreement statistics against a human gold standard and
prediction-powered inference (Angelopoulos et al. 2023)
<doi:10.1126/science.adi6000> with the power-tuned estimator of
Angelopoulos, Bates and Jordan (2023) <doi:10.48550/arXiv.2311.01453>, so a
small set of human labels debiases a large set of judge scores. Leaderboards
are supported through bootstrap rank intervals and Bradley-Terry ratings
(Bradley and Terry 1952) <doi:10.2307/2334029>. Accepts scores from any
evaluation harness.
| Version: |
0.1.0 |
| Depends: |
R (≥ 4.1.0) |
| Imports: |
cli (≥ 3.6.0), graphics, grDevices, stats, utils |
| Suggests: |
testthat (≥ 3.0.0), knitr, rmarkdown, sandwich |
| Published: |
2026-09-15 |
| DOI: |
10.32614/CRAN.package.evaluatellm (may not be active yet) |
| Author: |
Charles Coverdale [aut, cre, cph] |
| Maintainer: |
Charles Coverdale <charlesfcoverdale at gmail.com> |
| BugReports: |
https://github.com/charlescoverdale/evaluatellm/issues |
| License: |
MIT + file LICENSE |
| URL: |
https://charlescoverdale.github.io/evaluatellm/,
https://github.com/charlescoverdale/evaluatellm |
| NeedsCompilation: |
no |
| Language: |
en-GB |
| Materials: |
README, NEWS |
| CRAN checks: |
evaluatellm results |
Documentation:
Downloads:
Linking:
Please use the canonical form
https://CRAN.R-project.org/package=evaluatellm
to link to this page.