[R-meta] Questions regarding REML and FE models and R^2 calculation in metafor

[James Pustejovsky]

Whoops! I just posted the paper last Friday and it looks like it is still
waiting for moderation. Try this link:
https://osf.io/pdf2y

On Tue, Jul 25, 2023 at 2:03 PM Nevo Sagi <nevosagi8 using gmail.com> wrote:

> Thanks for the detailed explanation.
> The link you included doesn't work for me. Is there another way to get to
> that source?
>
>
> בתאריך יום ג׳, 25 ביולי 2023, 21:37, מאת James Pustejovsky ‏<
> jepusto using gmail.com>:
>
>> Hi Nevo,
>>
>> Responses inline below.
>>
>> Kind Regards,
>> James
>>
>> On Tue, Jul 25, 2023 at 1:37 AM Nevo Sagi <nevosagi8 using gmail.com> wrote:
>>
>>> I don't understand the rationale of using random effects at the
>>> experiment level. Experiments in my meta-analysis are parallel to
>>> observations in a conventional statistical analysis.
>>>
>>
>> I think this analogy doesn't follow. Conventional statistical analysis
>> does have observation-level error terms (i.e., level-1 error)--it's just
>> included by default as part of the model. In meta-analytic models, these
>> errors are not included unless explicitly specified.
>>
>>
>>> What is the meaning of using random effects at the observation level?
>>>
>>
>> Observation-level random effects here are used to capture heterogeneity
>> of effects across the experiments nested within a study. Considering that
>> you're interested in looking at moderators that vary across the experiments
>> reported in the same reference, it seems useful to attend to heterogeneity
>> at this level as well.
>>
>>
>>> In my understanding, by using random effects at the Reference level, I
>>> already tell the model to look at within-reference variation.
>>>
>>
>> This is not correct. Including reference-level random effects captures
>> _between-reference_ variation (or heterogeneity) of effects.
>>
>>
>>> In fact, the reason I was thinking to omit the random effect is because
>>> the model was over-sensitive to variation in effect size across moderator
>>> levels within specific references, while I am more interested in the total
>>> variation across the whole moderator spectrum, and therefore I want to
>>> focus more on the between-reference variation.
>>> Does that make sense?
>>>
>>
>> I stand by my original recommendation to consider including
>> experiment-level heterogeneity here. Omitting the experiment-level
>> heterogeneity more-or-less corresponds to averaging the effect size
>> estimates together so that you have one effect per reference, which will
>> tend to conceal within-reference heterogeneity. In fact, if you are using a
>> model that does not include moderators / predictors that vary at the
>> experiment level (within reference), then the correspondence is exact.
>> Further details here: https://osf.io/preprints/metaarxiv/pw54r/
>>
>

	[[alternative HTML version deleted]]