[R-meta] Unrealistic confidence limits for heterogeneity?

[Michael Dewey]

Just a few comments

1 - when you say negative estimates of variance do you mean of tau^2? If 
so it is perfectly possible to get negative estimates of variance by 
choosing an appropriate estimator of tau^2.

2 - the Q-profile method of estimating conifdnce intervals about the 
estimate of tau^2 is explained in references in the documentation, 
probably here Jackson, D., Turner, R., Rhodes, K., & Viechtbauer, W. 
(2014). Methods for calculating confidence and credible intervals for 
the residual between-study variance in random effects meta-regression 
models. BMC Medical Research Methodology, 14, 103. 
⁠https://doi.org/10.1186/1471-2288-14-103⁠

3 - I am not an expert on fitting mixed models in general in R but you 
need to specify which package you are using as there are a number of 
options, see the Task View https://cran.r-project.org/view=MixedModels
for more details.

Michael

On 30/03/2023 04:34, Will Hopkins via R-sig-meta-analysis wrote:
> No replies from anyone as yet, so here is some additional info, another
> question, and further thoughts about negative variance arising from multiple
> within-study effects measured in the same subjects.
> 
> The documentation for the way SAS generates confidence limits for variances
> and covariances in its mixed model is at this link:
> https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/statug/statug_mixed_s
> yntax01.htm#statug.mixed.procstmt_cl . They are Wald limits for the
> variances when negative variance is allowed. It doesn't state there how the
> standard errors for the variances are estimated, but further down on that
> page under COVTEST (an option you have to select) it refers to "asymptotic
> standard errors".
> 
> I asked in my previous message if the mixed models in R allow negative
> variance yet.  I forgot to ask if they provide standard errors yet, too.
> Again, they weren't provided a few years ago. Someone who was helping me
> with R found some code that someone had written to generate standard errors,
> but the estimates were different from those of SAS, so I gave up on R for
> mixed modeling at that point.
> 
> Further to multiple effects coming from the same subjects within studies
> resulting in negative variance for within-study heterogeneity... Multiple
> effects from the same subjects would be typically effects of multiple
> treatments in crossovers or effects of a treatment at different time points
> in a controlled trial, and they would be included in a meta-regression with
> appropriate fixed effects. The random error of measurement in such designs
> would not be correlated, but they would be correlated, if individual
> responses made substantial contributions to the measurement errors, and if
> the individual responses had some consistency across treatments or time
> point. In that case, the scatter of the means for each treatment or time
> point within each study, after fixed effects have been accounted for in the
> meta-regression, would be less than expected, given the standard error of
> the mean of each treatment or time point, so the variance would have to be
> negative (assuming there was no other source of within-study heterogeneity
> to offset the negative variance). In other words, negative variance would be
> evidence of consistent individual responses. In fact, if you change the sign
> and take the square root, I think you get an estimate of the consistent
> individual responses as a standard deviation. It's a lower limit for the
> individual responses, though, because there may be substantial positive
> unexplained within-study heterogeneity offsetting the negative variance.
> Also, I stated "I presume that combining the within- and between-study
> variances gives a realistic estimate of between-study heterogeneity, even
> when the within-study variance is negative." I'm probably wrong there: if
> you did separate metas for each treatment or time point, you would get
> correct estimates of the between-study heterogeneity, but I suspect that the
> average of those would be more than the sum of the positive between-study
> variance and negative within-study variance in the full meta model with two
> random effects, when there are consistent individual responses. I need to do
> some more simulations to check. I think the full model would give the most
> precise estimate of the mean effect, and correct precision for the modifiers
> in the model, because (hopefully) negative within-study variance correctly
> accounts for the repeated measurements on the same subjects.
> 
> Will Hopkins
> https://sportsci.org
> https://sportsci.org/will
> 
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> 

-- 
Michael
http://www.dewey.myzen.co.uk/home.html