[R-meta] Robust variance estimation

[James Pustejovsky]

Hi Emily,

I think one useful intuition about robust variance estimation is that its a
way of capturing the uncertainty in an estimate *using only between-study
variation*.

The RVE approach is analogous to how one would calculate the standard error
of a mean from a simple random sample, so it's helpful to review that
first. Say that we have a random sample of N observations, Y1, Y2,...., YN,
and we're trying to estimate the population mean from this sample. The
usual estimator is the sample mean, Y-bar. The standard error of Y-bar is
SE = SD / sqrt(N), where SD is the standard deviation of the sample
observations.

Okay so now let's think about the meta-analysis context. Say that you have
a meta-analysis with multiple effect sizes, which could be correlated,
nested within a set of studies, which can be treated as independent of each
other. And say that our goal is just to estimate the population mean effect
size across the set of studies.

The alternative to RVE is to use a "model-based" approach to uncertainty
estimation (like a multi-variate or hierarchical model). To do that
properly, we have to come up with an appropriate model for how the effect
sizes are related to each other (i.e., how they correlate) within each
study, and then also how they vary across studies. In other words, we have
to have a model for both the within-study variation (and covariation) and
the between-study variation. We use this model for the within- and
between-study variation to determine how to take a weighted average of all
of the effect sizes. And then, in the model-based approach, we also use it
to determine a standard error for the weighted average. As a result, the
accuracy of the standard error *is contingent on the modeling assumptions
being appropriate*.

The RVE approach still uses a model to determine how to take a weighted
average of all of the effect sizes, but it does not rely on the model for
assessing the uncertainty of the average. Instead, it just uses the between
study variation. Behind the RVE formulas are really two steps of
calculation. First is to calculate an average effect size for each study.
Since studies are independent, each of these average effects can be treated
as independent. And the overall average is just an average of the
study-specific average effect sizes (albeit with weights involved). So
actually, we're in a situation that's very similar to taking the mean of a
simple random sample, only now our "observations" are study-specific
average effect size estimates. Consequently, the second step in the RVE
standard error calculation is to take the SD of the study-specific average
effect sizes, then dividing the square root of the number of studies.
(Again, in practice there's weights involved, but the intuition is still
the same.)

There are two key advantage of this approach. One is that it works fine for
most any set of weights we might use in calculating the overall average
effect size. The weights don't have to be exactly right or optimal in any
sense. The second is that we can do these calculations without knowing
exactly how the individual effect sizes within each study are correlated
with each other. All we need is to be able to calculate study-specific
average effect sizes. So we don't have to rely on our modeling assumptions
being exactly right/accurate in order to trust the standard errors from
RVE.

The intuition about using only between-study variation can actually be
carried further to more complex scenarios with meta-regression on a set of
covariates, too.

Kind Regards,
James

On Fri, Apr 17, 2020 at 5:54 AM Emily Russell <emilyrussell99 using outlook.com>
wrote:

> Dear Friends and Colleagues
>
> I hope this is not too basic a question; but could someone give me an
> intuitive rather than technical explanation of what robust variance
> estimation does (as in robu in robumeta and robust in metafor)?  I have
> looked at the papers referred to but they are a bit 'heavy' for me.
>
> Thank you so much
>
> Emily
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]