[R-meta] Questions regarding REML and FE models and R^2 calculation in metafor

[James Pustejovsky]

Hi Nevo,

Responses inline below.

Kind Regards,
James

On Tue, Jul 25, 2023 at 1:37 AM Nevo Sagi <nevosagi8 using gmail.com> wrote:

> I don't understand the rationale of using random effects at the experiment
> level. Experiments in my meta-analysis are parallel to observations in a
> conventional statistical analysis.
>

I think this analogy doesn't follow. Conventional statistical analysis does
have observation-level error terms (i.e., level-1 error)--it's just
included by default as part of the model. In meta-analytic models, these
errors are not included unless explicitly specified.


> What is the meaning of using random effects at the observation level?
>

Observation-level random effects here are used to capture heterogeneity of
effects across the experiments nested within a study. Considering that
you're interested in looking at moderators that vary across the experiments
reported in the same reference, it seems useful to attend to heterogeneity
at this level as well.


> In my understanding, by using random effects at the Reference level, I
> already tell the model to look at within-reference variation.
>

This is not correct. Including reference-level random effects captures
_between-reference_ variation (or heterogeneity) of effects.


> In fact, the reason I was thinking to omit the random effect is because
> the model was over-sensitive to variation in effect size across moderator
> levels within specific references, while I am more interested in the total
> variation across the whole moderator spectrum, and therefore I want to
> focus more on the between-reference variation.
> Does that make sense?
>

I stand by my original recommendation to consider including
experiment-level heterogeneity here. Omitting the experiment-level
heterogeneity more-or-less corresponds to averaging the effect size
estimates together so that you have one effect per reference, which will
tend to conceal within-reference heterogeneity. In fact, if you are using a
model that does not include moderators / predictors that vary at the
experiment level (within reference), then the correspondence is exact.
Further details here: https://osf.io/preprints/metaarxiv/pw54r/

	[[alternative HTML version deleted]]