[R-meta] Unrealistic confidence limits for heterogeneity?

[James Pustejovsky]

Some further comments in addition to Michael's response below.

James

1. It is possible to allow for negative heterogeneity estimates using the
metafor package. Here is an example of the syntax:

library(metafor)

# generate data with no heterogeneity
set.seed(20230330)
k <- 10
vi <- 4 / (rpois(k, 22) + 8)
yi <- rnorm(k, mean = 0.2, sd = sqrt(vi))
dat <- data.frame(yi, vi)

# regular random effects meta-analysis, REML estimator
res1 <- rma(yi = yi, vi = vi, data=dat)
res1

# allow negative heterogeneity, REML estimator
res2 <- rma(yi = yi, vi = vi, data=dat, control=list(tau2.min=-min(vi)))
res2

# allow negative heterogeneity, other heterogeneity estimators
rma(yi = yi, vi = vi, data=dat, method = "ML",
control=list(tau2.min=-min(vi)))
rma(yi = yi, vi = vi, data=dat, method = "DL",
control=list(tau2.min=-min(vi)))
rma(yi = yi, vi = vi, data=dat, method = "HE",
control=list(tau2.min=-min(vi)))

2. You can obtain the estimated standard error for tau-squared as follows:
res1$se.tau2
res2$se.tau2

3. The metafor package implements several different confidence intervals
for tau-squared. The GENQ method requires estimating the model with method
GENQ.
confint(res1) # confidence interval for tau-squared
confint(res1, type = "PL") # profile likelihood method
confint(res1, type = "QP") # Q-profile method
rma(yi = yi, vi = vi, data=dat, weights = 1 / vi, method = "GENQ")  |>
  confint(type = "GENQ") # Generalized Q-statistic method

On Thu, Mar 30, 2023 at 5:32 AM Michael Dewey via R-sig-meta-analysis <
r-sig-meta-analysis using r-project.org> wrote:

> Just a few comments
>
> 1 - when you say negative estimates of variance do you mean of tau^2? If
> so it is perfectly possible to get negative estimates of variance by
> choosing an appropriate estimator of tau^2.
>
> 2 - the Q-profile method of estimating conifdnce intervals about the
> estimate of tau^2 is explained in references in the documentation,
> probably here Jackson, D., Turner, R., Rhodes, K., & Viechtbauer, W.
> (2014). Methods for calculating confidence and credible intervals for
> the residual between-study variance in random effects meta-regression
> models. BMC Medical Research Methodology, 14, 103.
> ⁠https://doi.org/10.1186/1471-2288-14-103⁠
>
> 3 - I am not an expert on fitting mixed models in general in R but you
> need to specify which package you are using as there are a number of
> options, see the Task View https://cran.r-project.org/view=MixedModels
> for more details.
>
> Michael
>
> On 30/03/2023 04:34, Will Hopkins via R-sig-meta-analysis wrote:
> > No replies from anyone as yet, so here is some additional info, another
> > question, and further thoughts about negative variance arising from
> multiple
> > within-study effects measured in the same subjects.
> >
> > The documentation for the way SAS generates confidence limits for
> variances
> > and covariances in its mixed model is at this link:
> >
> https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/statug/statug_mixed_s
> > yntax01.htm#statug.mixed.procstmt_cl . They are Wald limits for the
> > variances when negative variance is allowed. It doesn't state there how
> the
> > standard errors for the variances are estimated, but further down on that
> > page under COVTEST (an option you have to select) it refers to
> "asymptotic
> > standard errors".
> >
> > I asked in my previous message if the mixed models in R allow negative
> > variance yet.  I forgot to ask if they provide standard errors yet, too.
> > Again, they weren't provided a few years ago. Someone who was helping me
> > with R found some code that someone had written to generate standard
> errors,
> > but the estimates were different from those of SAS, so I gave up on R for
> > mixed modeling at that point.
> >
> > Further to multiple effects coming from the same subjects within studies
> > resulting in negative variance for within-study heterogeneity... Multiple
> > effects from the same subjects would be typically effects of multiple
> > treatments in crossovers or effects of a treatment at different time
> points
> > in a controlled trial, and they would be included in a meta-regression
> with
> > appropriate fixed effects. The random error of measurement in such
> designs
> > would not be correlated, but they would be correlated, if individual
> > responses made substantial contributions to the measurement errors, and
> if
> > the individual responses had some consistency across treatments or time
> > point. In that case, the scatter of the means for each treatment or time
> > point within each study, after fixed effects have been accounted for in
> the
> > meta-regression, would be less than expected, given the standard error of
> > the mean of each treatment or time point, so the variance would have to
> be
> > negative (assuming there was no other source of within-study
> heterogeneity
> > to offset the negative variance). In other words, negative variance
> would be
> > evidence of consistent individual responses. In fact, if you change the
> sign
> > and take the square root, I think you get an estimate of the consistent
> > individual responses as a standard deviation. It's a lower limit for the
> > individual responses, though, because there may be substantial positive
> > unexplained within-study heterogeneity offsetting the negative variance.
> > Also, I stated "I presume that combining the within- and between-study
> > variances gives a realistic estimate of between-study heterogeneity, even
> > when the within-study variance is negative." I'm probably wrong there: if
> > you did separate metas for each treatment or time point, you would get
> > correct estimates of the between-study heterogeneity, but I suspect that
> the
> > average of those would be more than the sum of the positive between-study
> > variance and negative within-study variance in the full meta model with
> two
> > random effects, when there are consistent individual responses. I need
> to do
> > some more simulations to check. I think the full model would give the
> most
> > precise estimate of the mean effect, and correct precision for the
> modifiers
> > in the model, because (hopefully) negative within-study variance
> correctly
> > accounts for the repeated measurements on the same subjects.
> >
> > Will Hopkins
> > https://sportsci.org
> > https://sportsci.org/will
> >
> > _______________________________________________
> > R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> > To manage your subscription to this mailing list, go to:
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >
>
> --
> Michael
> http://www.dewey.myzen.co.uk/home.html
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]