[R-meta] Getting meta-analytic residuals from multilevel models

[Viechtbauer, Wolfgang (SP)]

Hi Liam,

A generalization of the trim and fill method for meta-regression has been proposed:

https://onlinelibrary.wiley.com/doi/abs/10.1002/jrsm.1042

But this is not implemented in metafor and also doesn't apply to more complex models such as those that can be fitted with rma.mv().

Using the observed residuals for trim-and-fill (or some Egger-regression type of analysis) is problematic for various reasons:

1) The observed residuals are not independent. The sampling *errors* are independent (at least in the standard random/mixed-effects model -- in cases where multiple effect size estimates are computed based on the same subjects, then this is not true anymore either), but the observed *residuals* are not (despite what Nakagawa & Santos, 2011, write on page 1266).

2) The sampling variances of the residuals are not the same as the sampling variances of the observed effect sizes. So, if one were to use the residuals as input for further analyses, then it seems to me that one should really use the sampling variances of the residuals and not of the observed effect sizes.

3) I am not even sure what exactly Nakagawa & Santos (2011) really mean by eq. (41). I assume the notation here is based on models such as (6), (8), and (15). In those models, e_i denotes an estimate-level random effect and m_i the sampling error of a particular effect size estimate. Then they define the 'meta-analytic residual' as e_i + m_i. First of all, neither e_i nor m_i can be observed. One can compute BLUPs of the random effects and one can obtain the observed residuals, so the notation is unclear here. Also, I don't understand what their motivation is for adding together these two terms. And while I haven't worked through the math, I am fairly sure that the BLUPs are not independent either (and maybe not independent of the residuals) so now things are getting really messy.

I am not dismissing the idea outright, but I have reservations. Also, the trim-and-fill method has received some rather negative publicity in recent years. While I am not sure some of this is fully warranted, at least some simulation studies show that the method doesn't perform so well for detecting publication bias and for approximating the 'true effect' that is free of publication bias.

As for your actual question (at the end): The predict(), resid(), rstandard(), rstudent(), and ranef() functions in metafor all take the random effects structure of an rma.mv() model (however complex) into consideration. If you want residuals (defined in the usual sense of y_i - y-hat_i, where y-hat_i is the predicted effect based on the fixed effects), then resid() will give you that. rstandard() will also give you the standard errors thereof, which can be squared to obtain the sampling errors of the residuals. If you want the full var-cov matrix of the residuals, use vcov(<model>, type="resid") (which is not a diagonal matrix, since residuals are not independent). If you want BLUPs of the random effects, use ranef() (which also gives you SEs thereof).

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org]
>On Behalf Of Dougherty, Liam
>Sent: Friday, 24 April, 2020 15:13
>To: r-sig-meta-analysis using r-project.org
>Subject: [R-meta] Getting meta-analytic residuals from multilevel models
>
>Dear Wolfgang,
>
>I'm interested in testing for publication bias (e.g. non-publication of
>small studies) in a meta-analysis dataset with lots of non-independent data
>points (multiple measurements per study and per species). It seems clear
>that traditional graphical approaches are not appropriate here. One approach
>suggested by you (https://stats.stackexchange.com/questions/134768/metafor-
>package-in-r-ranktest-for-multivariate-meta-analysis) is to run a rma.mv
>model with study variance as a moderator. This seems to work well and give
>me a sensible result.
>
>However, if possible I would also like to use the trim-and-fill method, as
>it gives a nice graphical presentation of the problem. Nakagawa & Santos
>(2012; doi: 10.1007/s10682-012-9555-5) outline a method for this using meta-
>analytic residuals, and I've come across several papers using this approach,
>with residuals obtained from the MCMCglmm package (and code available to run
>this). However I don't think the predict function in metafor can be used to
>obtain residuals that take into account the random factors in a multilevel
>model- it looks to just use the mean estimate.
>
>So my question is: is there any way to obtain meta-analytic residuals from a
>multilevel model in Metafor? Or do I need to stick to MCMCglmm?
>
>Many thanks,
>Liam Dougherty
>
>-----------------------------------------------------
>
>Dr Liam R. Dougherty
>
>Early Career Research Fellow
>Department of Evolution, Ecology and Behaviour
>Institute of Integrative Biology
>University of Liverpool
>
>Biosciences Building | Room 236
>(+44) 0151 795 7771
>http://liamrdougherty.weebly.com/