[R-sig-ME] Cluster-robust SEs & random effects -- seeking some clarification

[James Pustejovsky]

Hi J.D.,
A few comments/reactions inline below.
James

On Wed, Jul 27, 2022 at 5:37 PM J.D. Haltigan <jhaltiga using gmail.com> wrote:

> ...
>
In the original investigation, the authors did not invoke a random effects
> model (but did use the pairIDs to control for fixed effects as noted and
> with robust SEs). Thus, in the original investigation there was *no*
> specification of a random effects model for the 'cluster' variable. We know
> from some other work there were some biases in village mapping and other
> possible sources of between-cluster variation that might be anticipated to
> have influence--at the random intercepts level--so we are looking into how
> specifying 'cluster' as a random effect might change the fixed effects
> estimates for the treatment intervention effect. In the Hamaker et al.
> language, it is indeed a 'random intercepts' only model.
>

I don't follow how using a random intercepts model improves the
generalizability warrant here. The random intercepts model is essentially
just a re-weighted average of the pair-specific effects in the original
analysis, where the weights are optimally efficient if the model is
correctly specified. That last clause carries a lot of weight here--correct
specification means 1) treatment assignment is unrelated to the random
effects, 2) the treatment effect is constant across clusters, 3)
distributional assumptions are valid (i.e., homoskedasticity at each level
of the model).

If the effects are heterogeneous, then I would think that including random
slopes on the treatment indicator would provide a better basis for
generalization. But even then, the warrant is still pretty vague---what is
the hypothetical population of villages from which the observed villages
are sampled?


> Given this, however, does it also make sense to include the cluster robust
> SEs for the fixed effects which would account for possible heterogeneity of
> treatment effects (i.e., slopes) across clusters?s
>
> If you're committed to the random intercepts model, then yes I think so
because using cluster robust SEs at least acknowledges the possibility of
heterogeneous treatment effects.


> Bottom line: in their original analyses, clusters are seen as
> interchangeable from a conceptual perspective (rather than drawn from a
> random universe of observations). When one scales up evidence to a universe
> of observations that are random (as they would be in the intended universe
> of inference in the real-world), then we are better positioned, I think, to
> adjudicate whether the mask intervention effect is 'practically
> significant' (in addition to whether the focal effect remains marginally
> significant from a frequentist perspective).
>
As noted above, this argument is a bit vague to me. If there's concern
about generalizability, then my first question would be: what is the target
population to which you are trying to generalize?

	[[alternative HTML version deleted]]