[R-meta] correct tau interpretation three-level meta-analysis

[James Pustejovsky]

Hi Filippo,

To add to Wolfgang's response, one note of caution regarding
interpreting the variance components in the three-level meta-analysis
(3LMA) model is that the variance component estimates are somewhat
sensitive to assumptions. If your data structure involves multiple,
correlated effect size estimates (i.e., estimates based on the same sample
of participants, so that the sampling errors of the estimates are
correlated), then the 3LMA model involves some degree of model
mis-specification. Currently available evidence suggests that the 3LMA may
be fairly robust with respect to inferences about *average effect
sizes*---that is, even though the model is mis-specified, hypothesis tests
and confidence intervals based on the model still have calibration rates
that are close to correct.

This robustness property does NOT extend to estimation of variance
components. If the model is mis-specified, then there will generally be
some degree of systematic bias in the variance component estimates. For
instance, say that the true correlation between effect size estimates from
the same sample is around r = 0.6. Using the 3LMA is equivalent to assuming
r = 0.0. As far as I understand, this will lead to estimates of
within-study heterogeneity that are systematically *too small* and
estimates of between-study heterogeneity that are systematically *too
large*. How strong the biases are depends on the structure of your data, so
it's hard to say much further here.

To your other question:

>
> Do we interpret it as an average variability within each cluster among
> clusters? Or we are assuming that each cluster has the same within-cluster
> variability?
>

I would say that the answer is "both." As formulated, the 3LMA model does
make the assumption that each cluster has the same within-cluster variance
component (i.e., homogeneity of variance within clusters). But even if this
assumption is incorrect, the estimated within-cluster variance will be some
sort of weighted average of the within-cluster variances, at least at an
approximate level. In principle, you could estimate cluster-specific
variances using the following (assuming that every value of outcome is
unique across studies):
```
rma.mv(yi, vi, random = list (~1|study, ~ study | outcome, struct = "DIAG")
```
But this probably isn't a good idea unless you have a lot of estimates from
every cluster. And the comments above regarding model mis-specification
apply here as well.

Kind Regards,
James

	[[alternative HTML version deleted]]