[R-meta] Unrealistic confidence limits for heterogeneity?

[Will Hopkins]

No replies from anyone as yet, so here is some additional info, another
question, and further thoughts about negative variance arising from multiple
within-study effects measured in the same subjects.

The documentation for the way SAS generates confidence limits for variances
and covariances in its mixed model is at this link:
https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.3/statug/statug_mixed_s
yntax01.htm#statug.mixed.procstmt_cl . They are Wald limits for the
variances when negative variance is allowed. It doesn't state there how the
standard errors for the variances are estimated, but further down on that
page under COVTEST (an option you have to select) it refers to "asymptotic
standard errors".

I asked in my previous message if the mixed models in R allow negative
variance yet.  I forgot to ask if they provide standard errors yet, too.
Again, they weren't provided a few years ago. Someone who was helping me
with R found some code that someone had written to generate standard errors,
but the estimates were different from those of SAS, so I gave up on R for
mixed modeling at that point.

Further to multiple effects coming from the same subjects within studies
resulting in negative variance for within-study heterogeneity... Multiple
effects from the same subjects would be typically effects of multiple
treatments in crossovers or effects of a treatment at different time points
in a controlled trial, and they would be included in a meta-regression with
appropriate fixed effects. The random error of measurement in such designs
would not be correlated, but they would be correlated, if individual
responses made substantial contributions to the measurement errors, and if
the individual responses had some consistency across treatments or time
point. In that case, the scatter of the means for each treatment or time
point within each study, after fixed effects have been accounted for in the
meta-regression, would be less than expected, given the standard error of
the mean of each treatment or time point, so the variance would have to be
negative (assuming there was no other source of within-study heterogeneity
to offset the negative variance). In other words, negative variance would be
evidence of consistent individual responses. In fact, if you change the sign
and take the square root, I think you get an estimate of the consistent
individual responses as a standard deviation. It's a lower limit for the
individual responses, though, because there may be substantial positive
unexplained within-study heterogeneity offsetting the negative variance.
Also, I stated "I presume that combining the within- and between-study
variances gives a realistic estimate of between-study heterogeneity, even
when the within-study variance is negative." I'm probably wrong there: if
you did separate metas for each treatment or time point, you would get
correct estimates of the between-study heterogeneity, but I suspect that the
average of those would be more than the sum of the positive between-study
variance and negative within-study variance in the full meta model with two
random effects, when there are consistent individual responses. I need to do
some more simulations to check. I think the full model would give the most
precise estimate of the mean effect, and correct precision for the modifiers
in the model, because (hopefully) negative within-study variance correctly
accounts for the repeated measurements on the same subjects.

Will Hopkins
https://sportsci.org
https://sportsci.org/will