Gtheory4LLM: Generalizability Theory for LLM Subjective Tasks
Studies the reliability and generalizability of subjective
judgments produced by large language models (LLMs), including annotation,
rating, and structured LLM-as-a-judge tasks. Specifies evaluator, prompt,
generation, and repeated-run facets through crossed or explicitly nested
random sources with configurable item interactions. Fits univariate models,
joint Gaussian models, and joint discrete models for binary, ordinal, and
unordered categorical outcomes, with source-specific covariance. Gaussian
models use exact balanced likelihood; discrete models use a dense
first-order Laplace approximation with Gaussian latent random effects.
Supported balanced decision studies compare evaluator, prompt, and
replication allocations using observed Gaussian or explicitly requested
latent binary and ordinal reliability, with random or fixed facets after
Brennan (2001). Gaussian fits report asymptotic Wald standard errors for
their variance components and delta-method intervals for the coefficients;
discrete fits report point estimates only. Scalar nominal reliability and
joint Gaussian-discrete fitting are not implemented. Includes three publicly
archived LLM annotation datasets covering hate-speech, mental-health,
and drug-review tasks. Discrete fitting is limited to small models; the
preflight report describes supported designs and computational limits.
Generalizability coefficients follow the variance-decomposition framework
of Brennan (2001) <doi:10.1007/978-1-4757-3456-0>.
| Version: |
0.2.0 |
| Depends: |
R (≥ 4.5.0) |
| Imports: |
Matrix (≥ 1.6.0), OpenMx (≥ 2.22.11), graphics, methods, stats, utils |
| Suggests: |
lme4, ordinal, knitr, rmarkdown |
| Published: |
2026-10-04 |
| DOI: |
10.32614/CRAN.package.Gtheory4LLM (may not be active yet) |
| Author: |
Jin Liu [aut, cre, cph] |
| Maintainer: |
Jin Liu <Veronica.Liu0206 at gmail.com> |
| BugReports: |
https://github.com/Veronica0206/Gtheory4LLM/issues |
| License: |
GPL-3 |
| URL: |
https://github.com/Veronica0206/Gtheory4LLM |
| NeedsCompilation: |
no |
| Citation: |
Gtheory4LLM citation info |
| Materials: |
README, NEWS |
| CRAN checks: |
Gtheory4LLM results |
Documentation:
Downloads:
Linking:
Please use the canonical form
https://CRAN.R-project.org/package=Gtheory4LLM
to link to this page.