[R-meta] Chi-square or F-test to test for subgroup heterogeneity

[James Pustejovsky]

The lmtest package includes functions coeftest and coefci that will let you
directly specify degrees of freedom for use in t-tests and CIs based on t
distributions. The alternative would be to calculate p-values/CIs manually.
Example below using the standard BCG vaccine data.

library(metafor)

# BCG data
dat.long <- to.long(measure="OR", ai=tpos, bi=tneg, ci=cpos, di=cneg,
data=dat.bcg)
levels(dat.long$group) <- c("exp", "con")
dat.long$group <- relevel(dat.long$group, ref="con")
dat.long <- escalc(measure="PLO", xi=out1, mi=out2, data=dat.long)

# bivariate random-effects model using rma.mv()
res <- rma.mv(yi, vi, mods = ~ group + alloc, random = ~ group | study,
struct="UN", data=dat.long)

# The model includes 13 independent studies and there are 3 predictors that
vary at the cluster level.
# Thus 10 degrees of freedom.

# calculate 95% CIs manually
b <- as.vector(res$beta)
se <- res$se
t_crit <- qt(0.975, df = 10)
b + t_crit * tcrossprod(se, c(-1,1))

# calculate p-values manually
(t_stat <- b / se)
(p_val <- 2 * pt(t_stat, df = 10))

library(lmtest)
coeftest(res, df = 10)
coefci(res, df = 10)


On Wed, Oct 10, 2018 at 7:13 PM Ty Beal <tbeal using gainhealth.org> wrote:

> Thank you, James,
>
>
>
> This is super helpful practical guidance. I like the approach of adjusting
> the degrees of freedom. Is there a simple way to specify the DF in the test
> via rma.mv()?
>
>
>
> Also, is the best reference to support this choice the Partlett et al.
> 2017 one that you listed?
>
>
>
> Thanks again,
>
> Ty
>
>
>
> *From: *James Pustejovsky <jepusto using gmail.com>
> *Date: *Tuesday, October 9, 2018 at 4:31 PM
> *To: *"Viechtbauer Wolfgang (SP)" <
> wolfgang.viechtbauer using maastrichtuniversity.nl>
> *Cc: *Ty Beal <tbeal using gainhealth.org>, "r-sig-meta-analysis using r-project.org"
> <r-sig-meta-analysis using r-project.org>
> *Subject: *Re: [R-meta] Chi-square or F-test to test for subgroup
> heterogeneity
>
>
>
> I agree with Wolfgang's assessment of the potential small-sample
> corrections in this situation. Satterthwaite or Kenward-Roger corrections
> should provide better type-I error control than z- or chi-squared tests,
> but I do not know of readily available tools for doing these calculations
> with rma.mv models. (Any students reading this, don't look now but a
> dissertation topic just fell into your lap!) However, I believe that
> Kenward-Roger is available in SAS. Partlett and Riley have examined the
> performance of KR corrections for univariate random effects meta-analysis,
> and Owens and Ferron have examined it for multi-level meta-analysis (in the
> context of meta-analysis of single-case designs). I do not know of research
> that has examined KR for multi-variate meta-analysis.
>
>
>
> In the absence of available tools, I think that it would be acceptable
> (and likely conservative) to use degrees of freedom for a t- or F- test
> equal to the number of independent clusters of effect sizes (typically the
> number of studies), minus the number of predictors in the model that vary
> between clusters. For example, say that you have a total of 20 studies, and
> you are testing for differences in average effect sizes across four
> categories (e.g., four regions). Then take the df to be 20 - 4 = 16.
>
>
>
> Another option would be to use tests based on robust variance estimation.
> There are robust versions of t- and F- tests that incorporate small-sample
> corrections and provide good type-I error control in relatively small
> samples. The tests are available in the clubSandwich package: t-tests using
> coef_test(), F-tests using Wald_test(). The F-tests tend to get
> conservative if you are testing a large number of moderators jointly (e.g.,
> testing equality among 6+ different categories). The drawback of this
> approach is that it is likely to be less powerful than using model-based
> variance estimates because it is based on a weaker set of assumptions. For
> instance, suppose again that you are testing for equality among four
> categories, A through D. The model-based variance estimator typically
> assumes that the between-study heterogeneity is constant across categories,
> whereas the robust variance estimator allows for different levels of
> heterogeneity in category A, category B, category C, and category D.
>
>
>
> James
>
>
>
> Owens, C. M., & Ferron, J. M. (2012). Synthesizing single-case studies: A
> Monte Carlo examination of a three-level meta-analytic model. *Behavior
> Research Methods*, *44*(3), 795-805.
>
>
>
> Partlett, C., & Riley, R. D. (2017). Random effects meta‐analysis:
> coverage performance of 95% confidence and prediction intervals following
> REML estimation. *Statistics in medicine*, *36*(2), 301-317.
>
>
>
>
>
> On Tue, Oct 9, 2018 at 10:18 AM Viechtbauer, Wolfgang (SP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
> Hi Ty,
>
> If we want to be picky, neither test="z" nor test="t" in rma.mv() is
> really justifiable. Using z- and chi-square tests ignores the uncertainty
> in the estimated variance components and can lead to inflated Type I error
> rates (but also overly conservative rates when there is very little or no
> heterogeneity).
>
> Using test="t" naively uses t- and F-tests with degrees of freedom equal
> to p and k-p dfs (where k is the total number of estimates and p the total
> number of model coefficients), but this is really an ad-hoc method -- that
> may indeed provide somewhat better control of the Type I error rates (at
> least when there is inflated to begin with), but again, the use of t- and
> F-distributions isn't properly motivated and the computation of the dfs is
> overly simplistic.
>
> The Knapp & Hartung method that is available for rma.uni() with
> test="knha" not only uses t- and F-tests, but also adjusts the standard
> errors in such a way that one actually gets t- and F-distributions under
> the null (technically, there is some fudging also involved in the K&H
> method, but numerous simulation studies have shown that this appears to be
> a non-issue).
>
> Unfortunately, test="knha" is not (currently) available for rma.mv(). A
> generalization of the K&H method to 'rma.mv' models is possible, but I
> have not implemented this so far, because further research is needed to
> determine if this is really useful.
>
> Another route would be to use t- and F-distribution, but then a
> Satterthwaite approximation to the dfs. I have examined this for rma.uni()
> models, but this appears to be overly conservative, especially under low
> heterogeneity. For moderate to large heterogeneity, this does appear to
> work though. Further research is also needed here to determine how well
> this would work for 'rma.mv' models. Also, working out how to implement
> this in general for 'rma.mv' models isn't trivial. The same applies to
> the method by Kenward and Roger.
>
> Maybe James (Pustejovsky) can also chime in here, since, together with
> Elizabeth Tipton, he has done some work on this topic when using
> cluster-robust inference methods.
>
> Best,
> Wolfgang
>
> -----Original Message-----
> From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On Behalf Of Ty Beal
> Sent: Friday, 05 October, 2018 21:03
> To: r-sig-meta-analysis using r-project.org
> Subject: [R-meta] Chi-square or F-test to test for subgroup heterogeneity
>
> Hi all,
>
> I estimated mean frequency of consumption as well as prevalence of
> less-than-daily fruit and vegetable consumption, at-least-daily carbonated
> beverage consumption, and at-least-weekly fast food consumption among
> school-going adolescents aged primarily 12-17 years from Africa, Asia,
> Oceania, and Latin America between 2008 and 2015. Random-effects
> meta-analysis was used to pool estimates globally and by WHO region, World
> Bank income group, and food system typology.
>
> To keep things simple, I will just ask about region. There are 5 regions
> included in the analysis. I would like to first test whether there is
> significant heterogeneity between all regions (omnibus test), and if so
> then do pairwise tests between specific regions. I am using rma.mv() with
> mods as the 5 regions and want to know whether I should use the default “z”
> statistic, which for the omnibus test is based on a chi-square distribution
> or “t”, which for the omnibus test is based on the F-distribution.
>
> Best,
>
> Ty Beal, PhD
> Technical Specialist
> Knowledge Leadership
>
> GAIN – Global Alliance for Improved Nutrition
> 1509 16th Street NW, 7th Floor | Washington, DC 20036
> tbeal using gainhealth.org<mailto:atumilowicz using gainhealth.org>
> C: +1 (602) 481-5211
> Skype: tyroniousbeal
> [GAINbanner]<http://www.gainhealth.org>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>
> [image: GAINbanner] <http://www.gainhealth.org>
>
>
> *Disclaimer*
>
> The information contained in this communication from the sender is
> confidential. It is intended solely for use by the recipient and others
> authorized to receive it. If you are not the recipient, you are hereby
> notified that any disclosure, copying, distribution or taking action in
> relation of the contents of this information is strictly prohibited and may
> be unlawful.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by *Mimecast Ltd*, an innovator in Software as a
> Service (SaaS) for business. Providing a *safer* and *more useful* place
> for your human generated data. Specializing in; Security, archiving and
> compliance. To find out more Click Here
> <http://www.mimecast.com/products/>.
>

	[[alternative HTML version deleted]]