[R-meta] Q-Statistic for Between-Subgroup Heterogeneity
Ty Beal
tbe@l @ending from g@inhe@lth@org
Thu Jul 26 19:15:36 CEST 2018
Dear Michael,
Thank you for your assistance. I used the same logic to choose a random effects meta-analysis.
Best,
Ty
-----Original Message-----
From: Michael Dewey <lists using dewey.myzen.co.uk>
Sent: Thursday, July 26, 2018 12:36 PM
To: Ty Beal <tbeal using gainhealth.org>; Viechtbauer, Wolfgang (SP) <wolfgang.viechtbauer using maastrichtuniversity.nl>; r-sig-meta-analysis using r-project.org
Subject: Re: [R-meta] Q-Statistic for Between-Subgroup Heterogeneity
Dear Ty
If theory suggests that tau^2 will vary then the model with separate
tau^2 seems preferable to me even if empirically in your particular case they do not. This would be similar to the choice between fixed and random effects which you make on the basis of theory not observed heterogeneity.
Michael
On 26/07/2018 00:03, Ty Beal wrote:
> Dear Wolfgang (and others),
>
> As a follow-up to the question of whether or not to pool tau^2 between subgroups, I am curious about using knowledge about the subject rather than the statistical model comparison to decide. Since I am doing a meta-analysis of prevalence (and not of traditional effect sizes) and do expect the true study-to-study dispersion to vary, wouldn't it be better to use different tau^2 values between groups even though the model comparison suggests I use a common tau^2?
>
> See the following quote from (https://www.meta-analysis.com/downloads/Meta-analysis%20Subgroups%20analysis.pdf)"
> "The decision to pool (or not) depends on the following. If we assume that the true study-to-study dispersion is the same within all subgroups, then observed differences in T2 must be due to sampling variation alone. In this case, we should pool the information to yield a common estimate, and then apply this estimate to all subgroups. This seems like a plausible expectation in the running example, where the study-to-study variation in effect size is likely to be similar for subgroups A and B.
>
> On the other hand, if we anticipate that the true between-studies dispersion may actually differ from one subgroup to the next, then we would estimate 2 within subgroups and use a separate estimate of 2 for each subgroup. For example, suppose that we are assessing an intervention to reduce recidivism among juvenile delinquents, and comparing the effect in subgroups of studies where the delinquents did, or did not, have a history of violence. We might expect to see a wider range of effect sizes in one subgroup than the other."
>
> Best,
> Ty
>
> On 7/25/18, 4:42 PM, "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> Yes, the model that assumes a common tau^2 value across groups is the simpler one.
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: Ty Beal [mailto:tbeal using gainhealth.org]
> Sent: Wednesday, 25 July, 2018 22:11
> To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
> Subject: Re: [R-meta] Q-Statistic for Between-Subgroup
> Heterogeneity
>
> Thanks, will post in plain text from now on. By simpler model, you mean the model with fixed tau^2? Just want to be sure.
>
> On 7/25/18, 4:06 PM, "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> Indeed, no sig. evidence for heteroscedasticity in the tau^2 values. Also, all of the information criteria (AIC, BIC, AICc) favor the simpler model.
>
> (please post in plain text -- then the formatting won't get
> messed up)
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: Ty Beal [mailto:tbeal using gainhealth.org]
> Sent: Wednesday, 25 July, 2018 21:57
> To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
> Subject: Re: [R-meta] Q-Statistic for Between-Subgroup
> Heterogeneity
>
> Sorry, the text did not format correctly. The pval is 0.3519.
>
> On 7/25/18, 3:54 PM, "Ty Beal" <tbeal using gainhealth.org> wrote:
>
> Thank you, Wolfgang. That clears things up.
>
> When I run the anova to compare models with fixed t^2 and variable t^2 I get the following:
>
> df AIC BIC AICc logLik LRT pval QE
> Full 10 -101.7226 -79.6756 -97.7940 60.8613 8083.0222
> Reduced 6 -105.3006 -92.0724 -103.9006 58.6503 4.4219 0.3519
> 8083.0222
>
> I'm still not sure how to decide between models, but if I go by the P-value, there is no clear evidence that using the variable t^2 is appropriate, right?
>
> Best,
> Ty
>
> On 7/25/18, 3:19 PM, "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> Hi Ty,
>
> Indeed, the test for between-group differences does not depend on which level of the factor is chosen as the reference level. For example:
>
> library(metafor)
>
> dat <- get(data(dat.bcg))
> dat <- escalc(measure="RR", ai=tpos, bi=tneg, ci=cpos, di=cneg, data=dat.bcg)
> res <- rma(yi, vi, mods = ~ alloc, data=dat)
> res
>
> # Test of Moderators (coefficients 2:3):
> # QM(df = 2) = 1.7675, p-val = 0.4132
>
> res <- rma(yi, vi, mods = ~ relevel(factor(alloc), ref="random"), data=dat)
> res
>
> # Test of Moderators (coefficients 2:3):
> # QM(df = 2) = 1.7675, p-val = 0.4132
>
> As for your question what happens when you use '-1':
>
> res <- rma(yi, vi, mods = ~ alloc - 1, data=dat)
> res
>
> # Test of Moderators (coefficients 1:3):
> # QM(df = 3) = 15.9842, p-val = 0.0011
>
> This is testing whether the average true outcome is equal to 0 for all levels. That is something entirely different than testing if there are between-group differences.
>
> As for pooling vs. not pooling: The number 5 is arbitrary and not a hard threshold. With few studies, the estimate of tau^2 is imprecise. You can always do a model comparison: Fit the model with common tau^2 and the model with different tau^2 values per subgroup and then do a likelihood ratio test with anova():
>
> res1 <- rma.mv(yi, vi, mods = ~ alloc, random = ~ 1 | trial, data=dat)
> res1
>
> res2 <- rma.mv(yi, vi, mods = ~ alloc, random = ~ alloc | trial, struct="DIAG", data=dat)
> res2
>
> anova(res1, res2)
>
> In this case, there is no significant evidence for heteroscedasticity in the tau^2 values across subgroups (p = 0.8521).
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: Ty Beal [mailto:tbeal using gainhealth.org]
> Sent: Wednesday, 25 July, 2018 17:41
> To: Viechtbauer, Wolfgang (SP); r-sig-meta-analysis using r-project.org
> Subject: Re: [R-meta] Q-Statistic for Between-Subgroup
> Heterogeneity
>
> Update.
>
> I read through Wolfgang's post here:
> http://www.metafor-project.org/doku.php/tips:multiple_factors_interact
> ions
>
> I realized that when I changed the reference level to a different region, the QM statistic and P-value remains the same. The only thing that changes is the coefficients.
>
> I am still not sure what is being tested when I do the subgroup - 1, but I think it is not appropriate for my problem.
>
> Additionally, after reading Wolfgang's other post (http://www.metafor-project.org/doku.php/tips:comp_two_independent_estimates), I am now using rma.mv to allow a different tau^2 in each of the five regions.
>
> I was trying to decide whether or not to pool tau^2 and it's certainly ideal to not pool in this context, since I except the variance between regions to be different. I read general advice that calculating a separate tau^2 for each subgroup is not recommended if the subgroup contains 5 or less studies (https://www.meta-analysis.com/downloads/Meta-analysis%20Subgroups%20analysis.pdf). For two out of four questions (there will be 4 meta-analyses in total) one of the regions only has 5 studies. Does anyone know whether it would it be better to pool estimates of tau^2 or not?
>
> Best,
> Ty
>
> On 7/24/18, 9:07 PM, "Ty Beal" <tbeal using gainhealth.org> wrote:
>
> Dear Wolfgang,
>
> Thanks for the tips. I am still getting used to the mailing list. I read through the thread and still have a couple questions, which will require me to explain my analysis in more detail.
>
> I am conducting a meta-analysis of prevalence for four dietary questions based on frequency of consumption. For example, the first question is coded so that I am calculating the prevalence that individuals consume fruit less than once per day. My data include 72 countries with nationally representative data that are weighted according to a complex survey design. This requires me to use the survey package to calculate the prevalence and SE/CIs for each of the 72 countries.
>
> I have done the meta-analysis for all the studies together and calculated an I^2 statistic. Now I am making a forest plot of subgroups (5 regions--and later also by sex). I would like to present some measure of between subgroup heterogeneity, but I am not sure the best approach to take. I had been using the following code to obtain a Q-statistic:
>
> fruit_res_r <- rma(yi, vi, method="DL", mods = ~ factor(region),
> data=fruit)
>
> But after reading the post that Wolfgang added, I realized that I am comparing the first region as the reference level. When I use factor(region) - 1, it produces much higher Q-statistics and lower P-values. I just want to be sure this is the best approach for this context.
>
> Best,
> Ty
>
> On 7/24/18, 5:58 PM, "Viechtbauer, Wolfgang (SP)" <wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> Dear Ty,
>
> Please always CC the mailing list.
>
> And please re-read the posts under the link I posted (i.e., the entire thread). In particular, this one is most relevant:
>
>
> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-December/00043
> 1.html
>
> It shows how to fit a model with different tau^2 values per subgroup using metafor. See also:
>
>
> http://www.metafor-project.org/doku.php/tips:comp_two_independent_esti
> mates
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: Ty Beal [mailto:tbeal using gainhealth.org]
> Sent: Tuesday, 24 July, 2018 17:18
> To: Viechtbauer, Wolfgang (SP)
> Subject: RE: [R-meta] Q-Statistic for Between-Subgroup
> Heterogeneity
>
> Dear Wolfgang,
>
> If you do end up updating metafor to allow different between-subgroup heterogeneity, please let me know (if you remember). I may want to re-run the analysis to see how big of a difference it makes in uncertainty estimates.
>
> Best,
> Ty
>
> -----Original Message-----
> From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf Of Viechtbauer, Wolfgang (SP)
> Sent: Monday, July 16, 2018 8:22 AM
> To: r-sig-meta-analysis using r-project.org
> Subject: Re: [R-meta] Q-Statistic for Between-Subgroup
> Heterogeneity
>
> Good point. This was discussed before (for metagen(), but it's the same issue):
>
>
> https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2017-December/00043
> 0.html
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Guido Schwarzer
> Sent: Monday, 16 July, 2018 13:13
> To: r-sig-meta-analysis using r-project.org
> Subject: Re: [R-meta] Q-Statistic for Between-Subgroup
> Heterogeneity
>
> Am 13.07.18 um 18:31 schrieb Viechtbauer, Wolfgang (SP):
>
> > [...]
> > In the output, you will find:
> >
> > Test of Moderators (coefficients 2:3):
> > QM(df = 2) = 1.7675, p-val = 0.4132
> >
> > This is the test for between-subgroup heterogeneity.
>
> ... assuming a common between-study variance tau2 across subgroups. The default in metaprop() of meta is to allow different between-study variances (analogous to RevMan 5, the software for Cochrane Reviews).
> However, this can be changed using argument 'tau.common = TRUE' in metaprop().
>
> As stated by Ty, metaprop() does not allow to specify the weights at the moment. I put this on my TODO list.
>
> Best wishes,
> Guido
>
>
> [GAINbanner]<http://www.gainhealth.org>
>
> Disclaimer
>
> The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
>
> This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
>
> [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
--
Michael
http://www.dewey.myzen.co.uk/home.html
[GAINbanner]<http://www.gainhealth.org>
Disclaimer
The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful.
This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast Ltd, an innovator in Software as a Service (SaaS) for business. Providing a safer and more useful place for your human generated data. Specializing in; Security, archiving and compliance. To find out more visit the Mimecast website.
[[alternative HTML version deleted]]
More information about the R-sig-meta-analysis
mailing list