[R-meta] Unrealistic confidence limits for heterogeneity?

[Will Hopkins]

James and Michael, thanks for your replies.  Yes, I meant negative values for the variance representing heterogeneity. Good to see that you can allow for such negative variance in metafor (and thanks for all that code, James). 

I'd still like to hear if negative variance is possible with the mixed model(s) in R, and whether it gives standard errors for the variances. I was trying out mixed modeling with lme4, as documented at https://www.sportsci.org/2016/inbrief.htm#SAS . There's a link there to some R programs and instructions provided by the person who was showing me how to use R, Alice Sweeting. It's in the paragraph below all the updates.

I still need feedback from anyone with an opinion on the argument for using negative variance, and feedback on the issue of the claim that there is downward bias in the estimate of heterogeneity. Someone called Donald Williams made that claim at https://psyarxiv.com/7tbrm/ : "with increasing values of tau^2, however, the DL estimator can show substantial downward bias... A number of studies have shown that the REML method shows less downward bias in the estimate of between study variance than the DL estimator (Chung, Rabe-Hesketh, & Choi, 2013; Sidik & Jonkman, 2007)". That's as may be, but the estimator in SAS is REML, and in my hands it gives unbiased estimates of the heterogeneity variance (tau^2), regardless of the number of studies, standard error of the study-estimates, and the true heterogeneity. Williams shows graphs only of tau, not tau^2. It  looks to me like the downward bias is simply small-DF bias, and you can correct it with the 1+1/(4DF) factor. Using Bayesian priors is not the way to correct it. If you want to make inferences about the magnitude of the heterogeneity relative to smallest and other important threshold values, square the threshold values and assume the variance has a t distribution, thereby avoiding the problem of small-DF bias in tau and partly offsetting the problem of the confidence interval being too narrow. (The threshold values for tau are one half the threshold values for the mean, by the way.)

I've started doing simulations of metas with multiple within-study effects from the same subjects in whom there are consistent individual responses. Yes, it does give negative variance for the within-study heterogeneity, but some of the individual-response variance adds to the between-study variance, which makes sense when you think about it. Also, the negative variance and the extra between-study variance are of the order of the individual response variance divided by the square root of the study sample size, so it's going to be a mess when sample sizes vary. Unfortunately the confidence interval for the mean effect is too wide, so for now, I can't do metas of multiple effects, if the effects include substantial consistent individual responses. I'm hoping it will be possible to somehow include and thereby adjust for the estimate of individual-response variance in each study (for a controlled trial, it's the difference in the variances of the change scores in the experimental and control groups), but I can't yet see how to do it. Maybe with some kind of spatial covariance structure.

Will
https://sportsci.org
https://sportsci.org/will

-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf Of James Pustejovsky via R-sig-meta-analysis
Sent: Friday, March 31, 2023 6:00 AM
To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-project.org>
Cc: James Pustejovsky <jepusto using gmail.com>
Subject: Re: [R-meta] Unrealistic confidence limits for heterogeneity?

Some further comments in addition to Michael's response below.

James

1. It is possible to allow for negative heterogeneity estimates using the metafor package. Here is an example of the syntax:

library(metafor)

# generate data with no heterogeneity
set.seed(20230330)
k <- 10
vi <- 4 / (rpois(k, 22) + 8)
yi <- rnorm(k, mean = 0.2, sd = sqrt(vi)) dat <- data.frame(yi, vi)

# regular random effects meta-analysis, REML estimator
res1 <- rma(yi = yi, vi = vi, data=dat)
res1

# allow negative heterogeneity, REML estimator
res2 <- rma(yi = yi, vi = vi, data=dat, control=list(tau2.min=-min(vi)))
res2

# allow negative heterogeneity, other heterogeneity estimators rma(yi = yi, vi = vi, data=dat, method = "ML",
control=list(tau2.min=-min(vi)))
rma(yi = yi, vi = vi, data=dat, method = "DL",
control=list(tau2.min=-min(vi)))
rma(yi = yi, vi = vi, data=dat, method = "HE",
control=list(tau2.min=-min(vi)))

2. You can obtain the estimated standard error for tau-squared as follows:
res1$se.tau2
res2$se.tau2

3. The metafor package implements several different confidence intervals for tau-squared. The GENQ method requires estimating the model with method GENQ.
confint(res1) # confidence interval for tau-squared confint(res1, type = "PL") # profile likelihood method confint(res1, type = "QP") # Q-profile method rma(yi = yi, vi = vi, data=dat, weights = 1 / vi, method = "GENQ")  |>
  confint(type = "GENQ") # Generalized Q-statistic method

On Thu, Mar 30, 2023 at 5:32 AM Michael Dewey via R-sig-meta-analysis < r-sig-meta-analysis using r-project.org> wrote:

> Just a few comments
>
> 1 - when you say negative estimates of variance do you mean of tau^2? 
> If so it is perfectly possible to get negative estimates of variance 
> by choosing an appropriate estimator of tau^2.
>
> 2 - the Q-profile method of estimating conifdnce intervals about the 
> estimate of tau^2 is explained in references in the documentation, 
> probably here Jackson, D., Turner, R., Rhodes, K., & Viechtbauer, W.
> (2014). Methods for calculating confidence and credible intervals for 
> the residual between-study variance in random effects meta-regression 
> models. BMC Medical Research Methodology, 14, 103.
> ⁠https://doi.org/10.1186/1471-2288-14-103⁠
>
> 3 - I am not an expert on fitting mixed models in general in R but you 
> need to specify which package you are using as there are a number of 
> options, see the Task View https://cran.r-project.org/view=MixedModels
> for more details.
>
> Michael
>
> On 30/03/2023 04:34, Will Hopkins via R-sig-meta-analysis wrote:
> > No replies from anyone as yet, so here is some additional info, 
> > another question, and further thoughts about negative variance 
> > arising from
> multiple
> > within-study effects measured in the same subjects.
> >
> > The documentation for the way SAS generates confidence limits for
> variances
> > and covariances in its mixed model is at this link:
> >
> https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/statug/statug_m
> ixed_s
> > yntax01.htm#statug.mixed.procstmt_cl . They are Wald limits for the 
> > variances when negative variance is allowed. It doesn't state there 
> > how
> the
> > standard errors for the variances are estimated, but further down on 
> > that page under COVTEST (an option you have to select) it refers to
> "asymptotic
> > standard errors".
> >
> > I asked in my previous message if the mixed models in R allow 
> > negative variance yet.  I forgot to ask if they provide standard errors yet, too.
> > Again, they weren't provided a few years ago. Someone who was 
> > helping me with R found some code that someone had written to 
> > generate standard
> errors,
> > but the estimates were different from those of SAS, so I gave up on 
> > R for mixed modeling at that point.
> >
> > Further to multiple effects coming from the same subjects within 
> > studies resulting in negative variance for within-study 
> > heterogeneity... Multiple effects from the same subjects would be 
> > typically effects of multiple treatments in crossovers or effects of 
> > a treatment at different time
> points
> > in a controlled trial, and they would be included in a 
> > meta-regression
> with
> > appropriate fixed effects. The random error of measurement in such
> designs
> > would not be correlated, but they would be correlated, if individual 
> > responses made substantial contributions to the measurement errors, 
> > and
> if
> > the individual responses had some consistency across treatments or 
> > time point. In that case, the scatter of the means for each 
> > treatment or time point within each study, after fixed effects have 
> > been accounted for in
> the
> > meta-regression, would be less than expected, given the standard 
> > error of the mean of each treatment or time point, so the variance 
> > would have to
> be
> > negative (assuming there was no other source of within-study
> heterogeneity
> > to offset the negative variance). In other words, negative variance
> would be
> > evidence of consistent individual responses. In fact, if you change 
> > the
> sign
> > and take the square root, I think you get an estimate of the 
> > consistent individual responses as a standard deviation. It's a 
> > lower limit for the individual responses, though, because there may 
> > be substantial positive unexplained within-study heterogeneity offsetting the negative variance.
> > Also, I stated "I presume that combining the within- and 
> > between-study variances gives a realistic estimate of between-study 
> > heterogeneity, even when the within-study variance is negative." I'm 
> > probably wrong there: if you did separate metas for each treatment 
> > or time point, you would get correct estimates of the between-study 
> > heterogeneity, but I suspect that
> the
> > average of those would be more than the sum of the positive 
> > between-study variance and negative within-study variance in the 
> > full meta model with
> two
> > random effects, when there are consistent individual responses. I 
> > need
> to do
> > some more simulations to check. I think the full model would give 
> > the
> most
> > precise estimate of the mean effect, and correct precision for the
> modifiers
> > in the model, because (hopefully) negative within-study variance
> correctly
> > accounts for the repeated measurements on the same subjects.
> >
> > Will Hopkins
> > https://sportsci.org
> > https://sportsci.org/will
> >
> > _______________________________________________
> > R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org 
> > To manage your subscription to this mailing list, go to:
> > https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> >
>
> --
> Michael
> http://www.dewey.myzen.co.uk/home.html
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org 
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]

_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org To manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis