[R-meta] residual heterogeneity in meta-regression

[Viechtbauer, Wolfgang (SP)]

Dear Daniel,

In my opinion, the best way to express how much heterogeneity there is is to compute a prediction interval for the true effects/outcomes. In its most basic form, this is computed for a random-effects model with:

mu_hat +- 1.96 * sqrt(tau^2_hat),

where mu_hat and tau^2_hat are just the estimates of mu (the average true effect/outcome) and tau^2 (the variance in the true effects/outcomes) and 1.96 is the 97.5% percentile of a standard normal distribution (rounded).

A slightly improved version takes the uncertainty of mu_hat into consideration:

mu_hat +- 1.96 * sqrt(SE[mu_hat]^2 + tau^2_hat),

where SE[mu] is the standard error of mu_hat. This is in fact how predict() computes this interval. See:

http://www.metafor-project.org/doku.php/faq#for_random-effects_models_fitt

If the 'Knapp and Hartung method' was used when fitting the model, instead of 1.96, the corresponding percentile of a t-distribution with df=k-1 is used.

This interval can be interpreted in various ways. It can be thought of as an interval that is meant to capture 95% of the true effects/outcomes (in the population of studies). It can also be thought of as an interval that is meant to capture the true effect/outcome of a new study (from the same population of studies) with 95% probability. This is very useful information, because it directly tells us how much true effects/outcomes can vary (in the actual units of the outcome/effect size measure). For example, it might be the case that mu_hat is positive and significantly different from 0 (i.e., true effects/outcomes are *on average* positive), but the prediction interval is so wide such that negative true effects/outcomes *in particular studies* are quite plausible.

I emphasize 'meant to' because, strictly speaking, the interval (computed in the forms above and also when using df=k-2) is not wide enough to really have a 95% capture probability. But for its intended purpose -- to provide a range that reflects how large/small the true effects/outcomes in particular studies might be -- it should be good enough.

For meta-regression models, one can compute such prediction intervals conditional on a particular combination of moderator values (e.g., for studies where mod1 = <something>, mod2 = <something>, ...).

Best,
Wolfgang

-----Original Message-----
From: Daniel Mønsted Shabanzadeh [mailto:dmshaban using gmail.com] 
Sent: Tuesday, 24 March, 2020 10:11
To: Viechtbauer, Wolfgang (SP)
Cc: r-sig-meta-analysis using r-project.org
Subject: Re: residual heterogeneity in meta-regression

Dear Wolfgang

Picking up the thread once more...Regarding the many studies included in the meta-regression model (up to 450 studies) I do expect very large Q statistics and cannot aim to explain all heterogeneity through meta-regression. Is there a better way to express heterogeneity in this case? I have read that the tau is a better way, however I am not sure of the interpretation and what I may conclude regarding the model and heterogeneity using tau. 

Regards,
Daniel  

On Mon, Nov 18, 2019 at 4:27 PM Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
Dear Daniel,

Roughly, the QE-test statistic should tend to decrease when R^2 is large. Whether this is strictly true depends on how tau^2 is being estimated. However, the QE-test could very well be significant even if R^2 is large. It simply means that there is still a significant amount of residual heterogeneity left.

Best,
Wolfgang

-----Original Message-----
From: Daniel Mønsted Shabanzadeh [mailto:dmshaban using gmail.com] 
Sent: Thursday, 14 November, 2019 11:09
To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
Subject: residual heterogeneity in meta-regression

Dear Wolfgang

I am performing a meta-regression on multiple one-arm non-randomised studies in order to explore the impact of moderators on complications following a surgical intervention. Adding moderators (age catrgory of the patient, surgical technique etc.) increases the R2:

Mixed-Effects Model (k = 183; tau^2 estimator: REML)

tau^2 (estimated amount of residual heterogeneity):     0.0074 (SE = 0.0010)
tau (square root of estimated tau^2 value):             0.0860
I^2 (residual heterogeneity / unaccounted variability): 99.20%
H^2 (unaccounted variability / sampling variability):   125.63
R^2 (amount of heterogeneity accounted for):            36.78%

Test for Residual Heterogeneity: 
QE(df = 156) = 4730.2255, p-val < .0001

Test of Moderators (coefficient(s) 2:27): 
QM(df = 26) = 115.5671, p-val < .0001

However, I am unaware of how to interpretate the rising R2 when QE tests keep on beeing significant. So far I understand that the rising R2 indicates that the heterogeneity is beeing partly explained by moderators, however does QE change in case most heterogeneity is explained by moderators?

Regards,
Daniel

Daniel Mønsted Shabanzadeh
MD, PhD
Department of Gastroenterology, Surgical Unit
Hvidovre Hospital
Mobile +45 2546 5251