[R-meta] Cooks distance from cluster in multilevel models

[Viechtbauer Wolfgang (SP)]

Cook's distance is a 'leave-out' measure. So, an estimate/cluster is removed, the model is refitted, and the change in the fixed effects (compared to the 'full' model based on all data and scaled by the var-cov matrix of the fixed effects from the full model) is quantified. It may not be possible to fit the model if a particular estimate/cluster is removed (e.g., the model doesn't converge anymore, the model matrix is then no longer of full rank). You should then see an NA value for that estimate/cluster.

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces at r-project.org] On Behalf Of Martineau, Roger
Sent: Friday, August 18, 2017 16:08
To: r-sig-meta-analysis at r-project.org
Subject: [R-meta] Cooks distance from cluster in multilevel models

Dear metafor users,

I downloaded the data set (i.e., p154-dataset.csv) from  Assinka and Wibbelink (2016) available from http://www.tqmp.org/RegularArticles/vol12-3/p154/p154.pdf

3-level model with multiple moderators (Listing 15 in the publication): 

> multiplemoderator <- rma.mv(y, v, mods = ~ pyear + typeovert + typecovert, 
+                             random =list(~ 1 | effectsizeID, ~ 1 | studyID), 
+                             tdist=TRUE, data=dataset)
> summary(multiplemoderator, digits=3)

Multivariate Meta-Analysis Model (k = 100; method: REML)

  logLik  Deviance       AIC       BIC      AICc  
 -63.375   126.750   138.750   154.136   139.694  

Variance Components: 

           estim   sqrt  nlvls  fixed        factor
sigma^2.1  0.085  0.292    100     no  effectsizeID
sigma^2.2  0.149  0.386     17     no       studyID

Test for Residual Heterogeneity: 
QE(df = 96) = 609.357, p-val < .001

Test of Moderators (coefficient(s) 2:4): 
F(df1 = 3, df2 = 96) = 6.414, p-val < .001

Model Results:

            estimate     se    tval   pval   ci.lb   ci.ub
intrcpt        0.466  0.107   4.346  <.001   0.253   0.678
pyear         -0.038  0.018  -2.077  0.040  -0.074  -0.002
typeovert     -0.204  0.139  -1.473  0.144  -0.479   0.071
typecovert    -0.709  0.191  -3.707  <.001  -1.089  -0.330

To plot Cook’s Distance values for cluster studyID (17 levels) from model multiple moderator, I did:

> #### Cook's distance ###
> par(mfrow=c(1,1))
> tmp.cook.studyID <- cooks.distance(multiplemoderator, 
+                            cluster=dataset$studyID, 
+                            progbar=TRUE)
  |=======================================================| 100%
> plot(tmp.cook.studyID, type="o", pch=19)
> which(tmp.cook.studyID > 1)
15 
15 

> which(tmp.cook.studyID > 0)
1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 17 
 1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 17 

Effect sizes are unique among studies. Study 16 is missing despite there are 16 effect sizes in study 16. Any clue for that?

Thanks in advance,

Roger ☺