[R-meta] Q-Statistic for Between-Subgroup Heterogeneity

[Viechtbauer, Wolfgang (SP)]

Indeed, no sig. evidence for heteroscedasticity in the tau^2 values. Also, all of the information criteria (AIC, BIC, AICc) favor the simpler model.

(please post in plain text -- then the formatting won't get messed up)

Best,
Wolfgang

-----Original Message-----
From: Ty Beal [mailto:tbeal using gainhealth.org] 
Sent: Wednesday, 25 July, 2018 21:57
To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity

Sorry, the text did not format correctly. The pval is 0.3519.

On 7/25/18, 3:54 PM, "Ty Beal" <tbeal using gainhealth.org> wrote:

Thank you, Wolfgang. That clears things up.

When I run the anova to compare models with fixed t^2 and variable t^2 I get the following:

        df       AIC      BIC      AICc  logLik    LRT   pval        QE
Full    10 -101.7226 -79.6756  -97.7940 60.8613               8083.0222
Reduced  6 -105.3006 -92.0724 -103.9006 58.6503 4.4219 0.3519 8083.0222

I'm still not sure how to decide between models, but if I go by the P-value, there is no clear evidence that using the variable t^2 is appropriate, right?

Best,
Ty

On 7/25/18, 3:19 PM, "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

Hi Ty,

Indeed, the test for between-group differences does not depend on which level of the factor is chosen as the reference level. For example:

library(metafor)

dat <- get(data(dat.bcg))
dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)
res <- rma(yi, vi, mods = ~ alloc, data=dat)
res

# Test of Moderators (coefficients 2:3):
# QM(df = 2) = 1.7675, p-val = 0.4132

res <- rma(yi, vi, mods = ~ relevel(factor(alloc), ref="random"), data=dat)
res

# Test of Moderators (coefficients 2:3):
# QM(df = 2) = 1.7675, p-val = 0.4132

As for your question what happens when you use '-1':

res <- rma(yi, vi, mods = ~ alloc - 1, data=dat)
res

# Test of Moderators (coefficients 1:3):
# QM(df = 3) = 15.9842, p-val = 0.0011

This is testing whether the average true outcome is equal to 0 for all levels. That is something entirely different than testing if there are between-group differences.

As for pooling vs. not pooling: The number 5 is arbitrary and not a hard threshold. With few studies, the estimate of tau^2 is imprecise. You can always do a model comparison: Fit the model with common tau^2 and the model with different tau^2 values per subgroup and then do a likelihood ratio test with anova():

res1 <- rma.mv(yi, vi, mods = ~ alloc, random = ~ 1 | trial, data=dat)
res1

res2 <- rma.mv(yi, vi, mods = ~ alloc, random = ~ alloc | trial, struct="DIAG", data=dat)
res2

anova(res1, res2)

In this case, there is no significant evidence for heteroscedasticity in the tau^2 values across subgroups (p = 0.8521).

Best,
Wolfgang

-----Original Message-----
From: Ty Beal [mailto:tbeal using gainhealth.org]
Sent: Wednesday, 25 July, 2018 17:41
To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity

Update.

I read through Wolfgang's post here: http://www.metafor-project.org/doku.php/tips:multiple_factors_interactions

I realized that when I changed the reference level to a different region, the QM statistic and P-value remains the same. The only thing that changes is the coefficients.

I am still not sure what is being tested when I do the subgroup - 1, but I think it is not appropriate for my problem.

Additionally, after reading Wolfgang's other post (http://www.metafor-project.org/doku.php/tips:comp_two_independent_estimates), I am now using rma.mv to allow a different tau^2 in each of the five regions.

I was trying to decide whether or not to pool tau^2 and it's certainly ideal to not pool in this context, since I except the variance between regions to be different. I read general advice that calculating a separate tau^2 for each subgroup is not recommended if the subgroup contains 5 or less studies (https://www.meta-analysis.com/downloads/Meta-analysis%20Subgroups%20analysis.pdf). For two out of four questions (there will be 4 meta-analyses in total) one of the regions only has 5 studies. Does anyone know whether it would it be better to pool estimates of tau^2 or not?

Best,
Ty

On 7/24/18, 9:07 PM, "Ty Beal" <tbeal using gainhealth.org> wrote:

Dear Wolfgang,

Thanks for the tips. I am still getting used to the mailing list. I read through the thread and still have a couple questions, which will require me to explain my analysis in more detail.

I am conducting a meta-analysis of prevalence for four dietary questions based on frequency of consumption. For example, the first question is coded so that I am calculating the prevalence that individuals consume fruit less than once per day. My data include 72 countries with nationally representative data that are weighted according to a complex survey design. This requires me to use the survey package to calculate the prevalence and SE/CIs for each of the 72 countries.

I have done the meta-analysis for all the studies together and calculated an I^2 statistic. Now I am making a forest plot of subgroups (5 regions--and later also by sex). I would like to present some measure of between subgroup heterogeneity, but I am not sure the best approach to take. I had been using the following code to obtain a Q-statistic:

fruit_res_r <- rma(yi, vi, method="DL", mods = ~ factor(region), data=fruit)

But after reading the post that Wolfgang added, I realized that I am comparing the first region as the reference level. When I use factor(region) - 1, it produces much higher Q-statistics and lower P-values. I just want to be sure this is the best approach for this context.

Best,
Ty

On 7/24/18, 5:58 PM, "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

Dear Ty,

Please always CC the mailing list.

And please re-read the posts under the link I posted (i.e., the entire thread). In particular, this one is most relevant:

https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-December/000431.html

It shows how to fit a model with different tau^2 values per subgroup using metafor. See also:

http://www.metafor-project.org/doku.php/tips:comp_two_independent_estimates

Best,
Wolfgang

-----Original Message-----
From: Ty Beal [mailto:tbeal using gainhealth.org]
Sent: Tuesday, 24 July, 2018 17:18
To: Viechtbauer, Wolfgang (SP)
Subject: RE: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity

Dear Wolfgang,

If you do end up updating metafor to allow different between-subgroup heterogeneity, please let me know (if you remember). I may want to re-run the analysis to see how big of a difference it makes in uncertainty estimates.

Best,
Ty

-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf Of Viechtbauer, Wolfgang (SP)
Sent: Monday, July 16, 2018 8:22 AM
To: r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity

Good point. This was discussed before (for metagen(), but it's the same issue):

https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-December/000430.html

Best,
Wolfgang

-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Guido Schwarzer
Sent: Monday, 16 July, 2018 13:13
To: r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity

Am 13.07.18 um 18:31 schrieb Viechtbauer, Wolfgang (SP):

> [...]
> In the output, you will find:
>
> Test of Moderators (coefficients 2:3):
> QM(df = 2) = 1.7675, p-val = 0.4132
>
> This is the test for between-subgroup heterogeneity.

... assuming a common between-study variance tau2 across subgroups. The default in metaprop() of meta is to allow different between-study variances (analogous to RevMan 5, the software for Cochrane Reviews).
However, this can be changed using argument 'tau.common = TRUE' in metaprop().

As stated by Ty, metaprop() does not allow to specify the weights at the moment. I put this on my TODO list.

Best wishes,
Guido