pairwiseLLM: Pairwise Comparison Tools for Large Language Model-Based Writing Evaluation
Provides a unified framework for generating, submitting, and analyzing pairwise comparisons of writing quality using large language models (LLMs). The package supports live and/or batch evaluation workflows across multiple providers ('OpenAI', 'Anthropic', 'Google Gemini', 'Together AI', and locally-hosted 'Ollama' models), includes bias-tested prompt templates and a flexible template registry, and offers tools for constructing forward and reversed comparison sets to analyze consistency and positional bias. Results can be modeled using Bradley–Terry (1952) <doi:10.2307/2334029> or Elo rating methods to derive writing quality scores. For information on the method of pairwise comparisons, see Thurstone (1927) <doi:10.1037/h0070288> and Heldsinger & Humphry (2010) <doi:10.1007/BF03216919>. For information on Elo ratings, see Clark et al. (2018) <doi:10.1371/journal.pone.0190393>.
| Version: | 1.1.0 |
| Depends: | R (≥ 4.1) |
| Imports: | curl, dplyr, httr2, jsonlite, rlang, stats, tibble, tidyselect, tools, utils |
| Suggests: | BradleyTerry2, EloChoice, knitr, mockery, purrr, readr, rmarkdown, sirt, stringr, testthat (≥ 3.0.0), tidyr, withr |
| Published: | 2025-12-22 |
| DOI: | 10.32614/CRAN.package.pairwiseLLM (may not be active yet) |
| Author: | Sterett H. Mercer 
[aut, cre] |
| Maintainer: | Sterett H. Mercer <sterett.mercer at ubc.ca> |
| BugReports: | https://github.com/shmercer/pairwiseLLM/issues |
| License: | MIT + file LICENSE |
| URL: | https://github.com/shmercer/pairwiseLLM, https://shmercer.github.io/pairwiseLLM/ |
| NeedsCompilation: | no |
| Citation: | pairwiseLLM citation info |
| Materials: | README, NEWS |
| CRAN checks: | pairwiseLLM results |
Documentation:
| Reference manual: | pairwiseLLM.html , pairwiseLLM.pdf |
| Vignettes: |
Advanced: Submitting and Polling Multiple Batches (source, R code) Getting Started with pairwiseLLM (source, R code) Prompt Template Positional Bias Testing (source, R code) |
Downloads:
| Package source: | pairwiseLLM_1.1.0.tar.gz |
| Windows binaries: | r-devel: not available, r-release: not available, r-oldrel: not available |
| macOS binaries: | r-release (arm64): not available, r-oldrel (arm64): not available, r-release (x86_64): not available, r-oldrel (x86_64): not available |
Linking:
Please use the canonical form https://CRAN.R-project.org/package=pairwiseLLM to link to this page.