[R-meta] robust variance estimation with small number of elements within the cluster

[James Pustejovsky]

Hi Diego,

clubSandwich uses cluster-robust (or "sandwich") variance estimators. These
estimators quantify the uncertainty in an average effect (or fixed effect
in a meta-regression model) *using only the between-cluster variation* in
effect sizes. Therefore, they simply will not work if all of the
information about a given category of effect sizes comes from a single
cluster.

As a rough heuristic, you can think of cluster-robust variance estimators
as involving the following. Consider your example where you have a
multi-level mixed model that has a categorical moderator and omits the
intercept. The model coefficients are therefore interpreted as average
effect sizes for each category of the moderator. Imagine aggregating the
effect sizes to the level of the study, so that there is only one average
effect size per category per study. Denote the aggregated effect for
category c in study j as T_cj. Suppose that there are k_c studies that
include effect sizes in category c. The overall average effect for category
c is then going to be, roughly, a weighted average of the study-level
aggregated effects for category c:

beta_c = sum_{j=1}^{k} w_cj T_cj

for some weights w_cj. (This is a rough approximation because, if you're
using a multilevel model, the estimate beta_c will actually also involve
the average effect sizes for categories other than c. But let's ignore that
wrinkle for purposes of building intuition.)

The robust estimator of the sampling variance of beta_c is going to be a
weighted version of the sample variance of the T_cj's:

V_c = sum_{j=1}^k (w_{cj})^2 (T_cj - beta_c)^2

If the weights are roughly equal, then the robust estimator ends up having
the even simpler form

V_c = S_c^2 / k_c,

where S_c^2 is the sample variance of the T_cj's. As you can see from this,
if there is only one study that includes effect size estimates in category
c, then S_c^2 will be zero by definition, but it can't be correct that
there is no uncertainty in the average effect size. If there's only one
observation, there's no information available to estimate the uncertainty
in the average. The cluster-robust variance estimator just won't work.

James

On Thu, Aug 26, 2021 at 3:36 AM Diego Grados Bedoya <diegogradosb using gmail.com>
wrote:

> Dear all,
>
> I would like to report the parameters obtained using a robust variance
> estimation of a multilevel mixed model (including a categorical moderator)
> without the intercept (sample size is 42). The categorical variable has 17
> levels of which 7 of them only have 1 study (due to the nature of the
> intervention). I am using as a cluster the study ID since some studies
> contributed with more than 1 effect size. Using the clubSandwhich R
> package, 4 out of the 7 levels of the moderator reported NA for the df and
> CIs values. I think the reason is that the estimates of these levels
> precisely coincide with the effect sizes obtained from the studies (se=0).
> I wonder if someone has faced something similar? (the robust function from
> metafor package is reporting the same values of the estimates).
>
> Any hint is more than welcome,
>
> Diego
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list
> R-sig-meta-analysis using r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]