[R-meta] possible miscalculation of Cook’s distances

[Antonina Dolgorukova]

Dear Dr. Viechtbauer and all,

I have a multilevel data structure (experiments nested within studies) are
use rma.mv to calculate an overall effect estimate. The next step requires
sensitivity analysis on the experiment-level data. For detecting outliers
I've used standardized (deleted) residuals and for detecting influential
experiments I've used Cook’s distance. However, the last test provides
contradictory results.

The forest plot indicates that the 1st experiment may be an outlier, and
standardized (deleted) residuals confirm this. But according to Cook’s
distance plot, the 2nd experiment is influential. It seems that there may
be a miscalculation of Cook’s distance since I can easily reproduce this
issue (in one study one of the experiments have to provide a much larger ES
than the other) also if use the model with random = ~ 1 | experiment, the
1st experiment is influential, not the second.

Could you please clarify is this a bug or a feature of the cooks.distance()
function? Maybe it does not work properly with rma.mv objects?

  ## Reproducible Example

# data frame
  dat <- data.frame(study=c(1,1,2,3,3,3), experiment=c(1:6),
                    yi=c( 68, 18, 31,20,10,26),
                    vi=c(60,32, 15, 19, 41, 82))

# multilevel model
res.ml <- rma.mv(yi, vi,
                 random = ~ 1 | study/experiment,
                 data=dat,
                 slab = paste("Study ", study,", ", "Experiment ",
experiment, sep = ""))

# forest plot examination  indicates that the 1st experiment may be an
outlier
forest(res.ml,
       header = "Study and Experiment ID")

# standardized (deleted) residuals confirm this
rst <- rstudent(res.ml)
plot(NA, NA, xlim=c(1, res.ml$k), ylim=c(-3,5),
     xlab="Study and Experiment ID", ylab="Standardized (Deleted) Residual",
     xaxt="n", las=1)
axis(side=1, at=1:res.ml$k, labels=rst$slab)
abline(h=c(-1.96,1.96), lty="dotted")
abline(h=0)
points(1:res.ml$k, rst$z, type="o", pch=19)

# according to Cook’s distance plot, the 2nd experiment is influential
cooksd <- cooks.distance(res.ml)
plot(1:length(cooksd), cooksd, ylim=c(0,2), type="o", pch=19, las=1,
xaxt="n",
     xlab="Experiment ID", ylab="Cook's Distance")
axis(1, 1:length(cooksd), labels=names(cooksd))
abline(h=4/length(cooksd), lty="dotted", lwd=2)

sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19042)
[1] metafor_3.1-43 metadat_1.0-0  Matrix_1.3-4

-- 
Antonina Dolgorukova, M.D.
Pavlov First Saint Petersburg State Medical University
Lev Tolstoy str., 6-8, 197022
Saint Petersburg, Russia

	[[alternative HTML version deleted]]