[R-meta] Q-Statistic for Between-Subgroup Heterogeneity
Viechtbauer, Wolfgang (SP)
wolfg@ng@viechtb@uer @ending from m@@@trichtuniver@ity@nl
Wed Jul 25 22:42:13 CEST 2018
Yes, the model that assumes a common tau^2 value across groups is the simpler one.
Best,
Wolfgang
-----Original Message-----
From: Ty Beal [mailto:tbeal using gainhealth.org]
Sent: Wednesday, 25 July, 2018 22:11
To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity
Thanks, will post in plain text from now on. By simpler model, you mean the model with fixed tau^2? Just want to be sure.
On 7/25/18, 4:06 PM, "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
Indeed, no sig. evidence for heteroscedasticity in the tau^2 values. Also, all of the information criteria (AIC, BIC, AICc) favor the simpler model.
(please post in plain text -- then the formatting won't get messed up)
Best,
Wolfgang
-----Original Message-----
From: Ty Beal [mailto:tbeal using gainhealth.org]
Sent: Wednesday, 25 July, 2018 21:57
To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity
Sorry, the text did not format correctly. The pval is 0.3519.
On 7/25/18, 3:54 PM, "Ty Beal" <tbeal using gainhealth.org> wrote:
Thank you, Wolfgang. That clears things up.
When I run the anova to compare models with fixed t^2 and variable t^2 I get the following:
df AIC BIC AICc logLik LRT pval QE
Full 10 -101.7226 -79.6756 -97.7940 60.8613 8083.0222
Reduced 6 -105.3006 -92.0724 -103.9006 58.6503 4.4219 0.3519 8083.0222
I'm still not sure how to decide between models, but if I go by the P-value, there is no clear evidence that using the variable t^2 is appropriate, right?
Best,
Ty
On 7/25/18, 3:19 PM, "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
Hi Ty,
Indeed, the test for between-group differences does not depend on which level of the factor is chosen as the reference level. For example:
library(metafor)
dat <- get(data(dat.bcg))
dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)
res <- rma(yi, vi, mods = ~ alloc, data=dat)
res
# Test of Moderators (coefficients 2:3):
# QM(df = 2) = 1.7675, p-val = 0.4132
res <- rma(yi, vi, mods = ~ relevel(factor(alloc), ref="random"), data=dat)
res
# Test of Moderators (coefficients 2:3):
# QM(df = 2) = 1.7675, p-val = 0.4132
As for your question what happens when you use '-1':
res <- rma(yi, vi, mods = ~ alloc - 1, data=dat)
res
# Test of Moderators (coefficients 1:3):
# QM(df = 3) = 15.9842, p-val = 0.0011
This is testing whether the average true outcome is equal to 0 for all levels. That is something entirely different than testing if there are between-group differences.
As for pooling vs. not pooling: The number 5 is arbitrary and not a hard threshold. With few studies, the estimate of tau^2 is imprecise. You can always do a model comparison: Fit the model with common tau^2 and the model with different tau^2 values per subgroup and then do a likelihood ratio test with anova():
res1 <- rma.mv(yi, vi, mods = ~ alloc, random = ~ 1 | trial, data=dat)
res1
res2 <- rma.mv(yi, vi, mods = ~ alloc, random = ~ alloc | trial, struct="DIAG", data=dat)
res2
anova(res1, res2)
In this case, there is no significant evidence for heteroscedasticity in the tau^2 values across subgroups (p = 0.8521).
Best,
Wolfgang
-----Original Message-----
From: Ty Beal [mailto:tbeal using gainhealth.org]
Sent: Wednesday, 25 July, 2018 17:41
To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity
Update.
I read through Wolfgang's post here: http://www.metafor-project.org/doku.php/tips:multiple_factors_interactions
I realized that when I changed the reference level to a different region, the QM statistic and P-value remains the same. The only thing that changes is the coefficients.
I am still not sure what is being tested when I do the subgroup - 1, but I think it is not appropriate for my problem.
Additionally, after reading Wolfgang's other post (http://www.metafor-project.org/doku.php/tips:comp_two_independent_estimates), I am now using rma.mv to allow a different tau^2 in each of the five regions.
I was trying to decide whether or not to pool tau^2 and it's certainly ideal to not pool in this context, since I except the variance between regions to be different. I read general advice that calculating a separate tau^2 for each subgroup is not recommended if the subgroup contains 5 or less studies (https://www.meta-analysis.com/downloads/Meta-analysis%20Subgroups%20analysis.pdf). For two out of four questions (there will be 4 meta-analyses in total) one of the regions only has 5 studies. Does anyone know whether it would it be better to pool estimates of tau^2 or not?
Best,
Ty
On 7/24/18, 9:07 PM, "Ty Beal" <tbeal using gainhealth.org> wrote:
Dear Wolfgang,
Thanks for the tips. I am still getting used to the mailing list. I read through the thread and still have a couple questions, which will require me to explain my analysis in more detail.
I am conducting a meta-analysis of prevalence for four dietary questions based on frequency of consumption. For example, the first question is coded so that I am calculating the prevalence that individuals consume fruit less than once per day. My data include 72 countries with nationally representative data that are weighted according to a complex survey design. This requires me to use the survey package to calculate the prevalence and SE/CIs for each of the 72 countries.
I have done the meta-analysis for all the studies together and calculated an I^2 statistic. Now I am making a forest plot of subgroups (5 regions--and later also by sex). I would like to present some measure of between subgroup heterogeneity, but I am not sure the best approach to take. I had been using the following code to obtain a Q-statistic:
fruit_res_r <- rma(yi, vi, method="DL", mods = ~ factor(region), data=fruit)
But after reading the post that Wolfgang added, I realized that I am comparing the first region as the reference level. When I use factor(region) - 1, it produces much higher Q-statistics and lower P-values. I just want to be sure this is the best approach for this context.
Best,
Ty
On 7/24/18, 5:58 PM, "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
Dear Ty,
Please always CC the mailing list.
And please re-read the posts under the link I posted (i.e., the entire thread). In particular, this one is most relevant:
https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-December/000431.html
It shows how to fit a model with different tau^2 values per subgroup using metafor. See also:
http://www.metafor-project.org/doku.php/tips:comp_two_independent_estimates
Best,
Wolfgang
-----Original Message-----
From: Ty Beal [mailto:tbeal using gainhealth.org]
Sent: Tuesday, 24 July, 2018 17:18
To: Viechtbauer, Wolfgang (SP)
Subject: RE: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity
Dear Wolfgang,
If you do end up updating metafor to allow different between-subgroup heterogeneity, please let me know (if you remember). I may want to re-run the analysis to see how big of a difference it makes in uncertainty estimates.
Best,
Ty
-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf Of Viechtbauer, Wolfgang (SP)
Sent: Monday, July 16, 2018 8:22 AM
To: r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity
Good point. This was discussed before (for metagen(), but it's the same issue):
https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-December/000430.html
Best,
Wolfgang
-----Original Message-----
From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Guido Schwarzer
Sent: Monday, 16 July, 2018 13:13
To: r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity
Am 13.07.18 um 18:31 schrieb Viechtbauer, Wolfgang (SP):
> [...]
> In the output, you will find:
>
> Test of Moderators (coefficients 2:3):
> QM(df = 2) = 1.7675, p-val = 0.4132
>
> This is the test for between-subgroup heterogeneity.
... assuming a common between-study variance tau2 across subgroups. The default in metaprop() of meta is to allow different between-study variances (analogous to RevMan 5, the software for Cochrane Reviews).
However, this can be changed using argument 'tau.common = TRUE' in metaprop().
As stated by Ty, metaprop() does not allow to specify the weights at the moment. I put this on my TODO list.
Best wishes,
Guido
More information about the R-sig-meta-analysis
mailing list