[R-meta] robust variance estimation with small number of elements within the cluster

[Diego Grados Bedoya]

Hi James,

Thank you very much for your nice explanation.

Greetings,

Diego

On Thu, 26 Aug 2021 at 18:11, James Pustejovsky <jepusto using gmail.com> wrote:

> Hi Diego,
>
> clubSandwich uses cluster-robust (or "sandwich") variance estimators.
> These estimators quantify the uncertainty in an average effect (or fixed
> effect in a meta-regression model) *using only the between-cluster
> variation* in effect sizes. Therefore, they simply will not work if all of
> the information about a given category of effect sizes comes from a single
> cluster.
>
> As a rough heuristic, you can think of cluster-robust variance estimators
> as involving the following. Consider your example where you have a
> multi-level mixed model that has a categorical moderator and omits the
> intercept. The model coefficients are therefore interpreted as average
> effect sizes for each category of the moderator. Imagine aggregating the
> effect sizes to the level of the study, so that there is only one average
> effect size per category per study. Denote the aggregated effect for
> category c in study j as T_cj. Suppose that there are k_c studies that
> include effect sizes in category c. The overall average effect for category
> c is then going to be, roughly, a weighted average of the study-level
> aggregated effects for category c:
>
> beta_c = sum_{j=1}^{k} w_cj T_cj
>
> for some weights w_cj. (This is a rough approximation because, if you're
> using a multilevel model, the estimate beta_c will actually also involve
> the average effect sizes for categories other than c. But let's ignore that
> wrinkle for purposes of building intuition.)
>
> The robust estimator of the sampling variance of beta_c is going to be a
> weighted version of the sample variance of the T_cj's:
>
> V_c = sum_{j=1}^k (w_{cj})^2 (T_cj - beta_c)^2
>
> If the weights are roughly equal, then the robust estimator ends up having
> the even simpler form
>
> V_c = S_c^2 / k_c,
>
> where S_c^2 is the sample variance of the T_cj's. As you can see from
> this, if there is only one study that includes effect size estimates in
> category c, then S_c^2 will be zero by definition, but it can't be correct
> that there is no uncertainty in the average effect size. If there's only
> one observation, there's no information available to estimate the
> uncertainty in the average. The cluster-robust variance estimator just
> won't work.
>
> James
>
> On Thu, Aug 26, 2021 at 3:36 AM Diego Grados Bedoya <
> diegogradosb using gmail.com> wrote:
>
>> Dear all,
>>
>> I would like to report the parameters obtained using a robust variance
>> estimation of a multilevel mixed model (including a categorical moderator)
>> without the intercept (sample size is 42). The categorical variable has 17
>> levels of which 7 of them only have 1 study (due to the nature of the
>> intervention). I am using as a cluster the study ID since some studies
>> contributed with more than 1 effect size. Using the clubSandwhich R
>> package, 4 out of the 7 levels of the moderator reported NA for the df and
>> CIs values. I think the reason is that the estimates of these levels
>> precisely coincide with the effect sizes obtained from the studies (se=0).
>> I wonder if someone has faced something similar? (the robust function from
>> metafor package is reporting the same values of the estimates).
>>
>> Any hint is more than welcome,
>>
>> Diego
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis using r-project.org
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>

	[[alternative HTML version deleted]]