[R-meta] Diagnostic tests meta regression

[Viechtbauer, Wolfgang (NP)]

Dear Adelina,

The standard random/mixed-effects model (or 'normal-normal model' as it is also sometimes called) makes two normality assumptions:

1) that each of the k sampling distributions is normal
2) that the random effects are normal

Assumption 1 says that if a particular study were to be repeated under identical circumstances (but including new subjects on each repetition) an infinite number of times, then the values of the chosen effect size measure that would be observed would form a normal distribution. Note that this is different from checking that the k observed effect sizes are normally distributed (which actually is not relevant, since the model doesn't assume that). One way of checking into this assumption is via simulation studies to see how quickly the central limit theorem 'kicks in' so that the sampling distribution is indeed approximately normal. But I have never seen anybody do this for a particular meta-analysis. Instead, we tend to rely on the collective wisdom about the properties of various effect size measures to have a rough sense whether assumption 1 is somewhat sensible. For example, a meta-analysis of raw correlation coefficients where sample sizes are smallish and the correlations tend to be large is something I would avoid, since we know that the sampling distribution of a raw correlation coefficient under such circumstances is not normal (instead, use r-to-z transformed values for the meta-analysis). Simarly, for effect size measures based on dichotomous variables, things don't look so good if the outcome of interest is rare (instead, consider 'exact' models based on the binomial distribution).

As for assumption 2: In principle, one could examine the distribution of the BLUPs (the predicted values of the random effects) to check whether this assumption is somewhat reasonable. However, it is unclear how well this actually works.

One could even argue that a QQ plot of the standardized/studentized residuals is kind of a check of assumptions 1 and 2 combined, but it is also quite unclear how well this actually works.

Also, checking the distribution of the BLUPs / standardized/studentized residuals for normality assumes that one has captured all relevant moderators and the absence of publication bias.

In general, this is just all super messy / difficult. Moreover, it isn't even all that clear how important it is to check these assumptions. For example, for estimating the pooled effect and its CI, the distribution of the random effects might not even be that relevant to begin with (but definitely for constructing a prediction interval).

So, in the end, there really isn't a simple answer here.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Adelina Artenie
>Sent: Thursday, 22 December, 2022 15:19
>To: r-sig-meta-analysis using r-project.org
>Subject: [R-meta] Diagnostic tests meta regression
>
>Hi Wolfgang/all,
>
>I am trying to check my model assumptions and I think I don't understand fully
>what to look at and how, so if possible, any advice you may have would be greatly
>appreciated:
>
>I am running a mixed-effects meta-regression model. Because the outcome (a rate)
>is rare in some cases, there is a concern that the normality assumption is not
>met (Viechtbauer, Chapter 11, Handbook of Meta Analysis).
>
>As I read about this, my understanding is that both the random-effects and the
>random-error terms are assumed to be normally distributed (though this is not
>100% clear to me, because different sources tend to focus on one term or the
>other, some<https://www.meta-analysis.com/downloads/MRManual.pdf> say that the
>method-of-moments does not, in fact, make assumptions about the distribution of
>random-effects (page 150)).
>
>If true that both terms have to be normally distributed, I am trying to
>understand how to visualise/test this. One straightforward approach would be to
>look at a normal QQplot.  My questions are:
>
>  1.  If I follow this approach<https://www.metafor-
>project.org/doku.php/plots:normal_qq_plots>, and run a mixed-effects meta-
>regression (res4), I would be visualising/checking the assumption that the
>residual heterogeneity in the true effects is normally distributed or not. Am I
>correct in saying that this would be a proxy for checking the assumption that the
>random-effects are normally distributed?
>
>  1.  If so, how would I check if the error terms are normally distributed? Would
>I plot a normal QQplot for each study? I have several outcomes and a total of 150
>outcomes..
>
>Thank you in advance,
>Adelina