[R-meta] robust error is smaller than model-based error

[James Pustejovsky]

Hi Yefeng,

On point 1, I am not sure what your question is. From inspecting the source
code of metafor::robust(), the function is not set up to handle models with
crossed random effects. I'm not at all sure what it does if you feed it a
model with crossed random effects, but I would be very cautious about
interpreting the output. Perhaps Wolfgang can comment on whether robust()
is meant to accommodate models with crossed random effects.

On point 2, I can verify that clubSandwich does not support CRVE for models
with crossed random effects. Cameron, Gelbach, and Miller (2011) describe
multi-way clustered standard errors, but only for ordinary least squares
models. As far as I am aware, the statistical theory for multi-way
clustered standard errors has not been developed for models that have
crossed random effects and the extension from Cameron, Gelbach and Miller
is not obvious. So if you want to stay on solid ground in terms of
statistical theory, I think your best approach might be just to do a good
job of developing and checking the model, and then rely on the model-based
SEs for inference.

James

On Thu, Feb 15, 2024 at 7:37 PM Yefeng Yang via R-sig-meta-analysis <
r-sig-meta-analysis using r-project.org> wrote:

> Dear community,
>
> I (or, more precisely, my collaborator) am helping with one meta-analysis
> with dependent effect sizes. We used a multilevel model with effect size
> ID, study ID, and species ID as random effects.  We also used the RVE to
> calculate the robust error. I have two questions.
>
>
>   1.
> The test of model coefficient based on RVE indicates a significant effect
> (p < 0.05), while the test based on model-based error (we call it
> naive/original error) shows a non-significant effect (p < 0.05). I used
> `robust` in `metafor`, with `CR1` correction (`clubsandwich` is not working
> in my case; see below​) . Sorry, I do not have the raw data so there is no
> reproducible example.
>   2.
> How to calculate the robust error for models with non-nested
> random-effects structure? This issue has troubled me for a long time.
> Precisely, in my case, because effect size ID is nested within the study
> ID, so it is easy to calculate robust error (either using ​ ​`robust` or
> ​`clubsandwich` ​). However, I still have species ID as the random effect
> (it is a kind of crossed random effect). In such a case `clubsandwich` is
> not working. `robust` is still working, but we only can use `CR1`
> correction.
>
> Regards,
> Yefeng
>
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>

	[[alternative HTML version deleted]]