# [R-meta] possible miscalculation of Cook’s distances

Antonina Dolgorukova @n@do|gorukov@ @end|ng |rom gm@||@com
Thu Feb 3 22:36:15 CET 2022

```Dear Dr. Viechtbauer and all,

I have a multilevel data structure (experiments nested within studies) are
use rma.mv to calculate an overall effect estimate. The next step requires
sensitivity analysis on the experiment-level data. For detecting outliers
I've used standardized (deleted) residuals and for detecting influential
experiments I've used Cook’s distance. However, the last test provides

The forest plot indicates that the 1st experiment may be an outlier, and
standardized (deleted) residuals confirm this. But according to Cook’s
distance plot, the 2nd experiment is influential. It seems that there may
be a miscalculation of Cook’s distance since I can easily reproduce this
issue (in one study one of the experiments have to provide a much larger ES
than the other) also if use the model with random = ~ 1 | experiment, the
1st experiment is influential, not the second.

Could you please clarify is this a bug or a feature of the cooks.distance()
function? Maybe it does not work properly with rma.mv objects?

## Reproducible Example

# data frame
dat <- data.frame(study=c(1,1,2,3,3,3), experiment=c(1:6),
yi=c( 68, 18, 31,20,10,26),
vi=c(60,32, 15, 19, 41, 82))

# multilevel model
res.ml <- rma.mv(yi, vi,
random = ~ 1 | study/experiment,
data=dat,
slab = paste("Study ", study,", ", "Experiment ",
experiment, sep = ""))

# forest plot examination  indicates that the 1st experiment may be an
outlier
forest(res.ml,
header = "Study and Experiment ID")

# standardized (deleted) residuals confirm this
rst <- rstudent(res.ml)
plot(NA, NA, xlim=c(1, res.ml\$k), ylim=c(-3,5),
xlab="Study and Experiment ID", ylab="Standardized (Deleted) Residual",
xaxt="n", las=1)
axis(side=1, at=1:res.ml\$k, labels=rst\$slab)
abline(h=c(-1.96,1.96), lty="dotted")
abline(h=0)
points(1:res.ml\$k, rst\$z, type="o", pch=19)

# according to Cook’s distance plot, the 2nd experiment is influential
cooksd <- cooks.distance(res.ml)
plot(1:length(cooksd), cooksd, ylim=c(0,2), type="o", pch=19, las=1,
xaxt="n",
xlab="Experiment ID", ylab="Cook's Distance")
axis(1, 1:length(cooksd), labels=names(cooksd))
abline(h=4/length(cooksd), lty="dotted", lwd=2)

sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19042)