[R-meta] Outlier and influence cases 3-level meta-analysis

[d@@iei@i@gucci@rdi m@iii@g oii gm@ii@com]

Thanks Wolfgang, appreciate the quick response and guidance. I was unaware
of the cluster feature in the 'devel' version of metafor, so this
availability will enable us to examine outliers at both levels - awesome!

Cheers,
Daniel

-----Original Message-----
From: Viechtbauer, Wolfgang (SP)
<wolfgang.viechtbauer using maastrichtuniversity.nl> 
Sent: Thursday, 18 March 2021 7:50 PM
To: daniel.f.gucciardi using gmail.com; r-sig-meta-analysis using r-project.org
Subject: RE: [R-meta] Outlier and influence cases 3-level meta-analysis

Dear Daniel,

No fudamental issues, except that I have not yet implemented all of this for
'rma.mv' objects. Assuming that you have installed the 'devel' version of
metafor (https://wviechtb.github.io/metafor/#installation), things like
rstandard(), rstudent(), cooks.distance(), and dfbetas() are however already
available to you, which are some of the main tools for detecting outliers
and/or influential cases anyway (covariance ratios would also be nice, but
not yet available). There is also hatvalues() and weights(), but these may
be a bit less accessible.

One issue in more complex models is the question at which 'level' we want to
detect outliers / influential cases. For rma.mv objects, things like
rstandard() and cooks.distance() have a 'cluster' argument, which can be
used to specify a grouping variable. For example:

dat <- dat.konstantopoulos2011
res <- rma.mv(yi, vi, random = ~ 1 | district/school, data=dat)
cooks.distance(res)

will compute the Cook's distances for each individual estimate, while

cooks.distance(res, cluster=dat$district)

will compute the Cook's distances for each level of 'district'. Or easier
for interpretation:

plot(cooks.distance(res), type="o", pch=19)

suggests maybe 2 somewhat influential estimates and

plot(cooks.distance(res, cluster=dat$district), type="o", pch=19)

one influential district (probably the one with those 2 estimates, but I
haven't checked).

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis 
>[mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of 
>daniel.f.gucciardi using gmail.com
>Sent: Wednesday, 17 March, 2021 9:44
>To: r-sig-meta-analysis using r-project.org
>Subject: [R-meta] Outlier and influence cases 3-level meta-analysis
>
>Hi all,
>
>My colleagues and I have conducted a 3-level meta-analysis using 
>metafor in which we account for multiple outcomes from individual 
>studies. As part of our sensitivity analyses, we examined outlier and 
>influential cases using the recommendations of Viechtbauer and Cheung 
>(https://onlinelibrary.wiley.com/doi/10.1002/jrsm.11), which seems to 
>be a common approach. Are there any issues in applying their framework 
>for two-level meta-analytic models to three-level models? If so, I'd 
>appreciate your direction to any literature that might help me 
>appreciate these issues please.
>
>Regards,
>
>Daniel