[R-meta] Questions regarding REML and FE models and R^2 calculation in metafor

[Nevo Sagi]

Hi James,
Thanks for the quick response.

Centering moderators by reference is an interesting idea. I will try it.

I don't understand the rationale of using random effects at the experiment
level. Experiments in my meta-analysis are parallel to observations in a
conventional statistical analysis. What is the meaning of using random
effects at the observation level?
In my understanding, by using random effects at the Reference level, I
already tell the model to look at within-reference variation. In fact, the
reason I was thinking to omit the random effect is because the model was
over-sensitive to variation in effect size across moderator levels within
specific references, while I am more interested in the total variation
across the whole moderator spectrum, and therefore I want to focus more on
the between-reference variation.
Does that make sense?


Best,
Nevo



On Mon, Jul 24, 2023 at 10:48 PM James Pustejovsky <jepusto using gmail.com>
wrote:

> Hi Nevo,
>
> Considering the structure of your data (50 references with an average of
> 10 experiments per  reference), I would suggest moving to a more flexible
> model that includes random effects not only at the level of reference, but
> also at the level of experiment, as in:
> random = ~ 1 | Reference / Experiment
> Using this random effects structure will then let you describe how the
> moderator explains variation both between references and within references
> (i.e., by comparing the variance components from a model with moderators to
> the variance components from a model with an intercept alone).
>
> It could also be useful to center the moderators by reference (i.e.,
> calculate the reference-specific mean of the moderator and then subtract
> this from the original values of the moderator). Centering is akin to
> de-composing the predictor into within-reference and between-reference
> variation. The within-reference variation would come only from those 7
> studies where the value of the moderator changes across experiments. The
> between-reference variation would come from all 50 studies if different
> articles use different levels of the moderator. The model for a moderator X
> would then be:
> modes = ~ X_mean + X_centered
> I would anticipate that the coefficients on these predictors would be less
> sensitive to the random effects specification than using the un-centered
> predictor X.
>
> James
>
>
> On Mon, Jul 24, 2023 at 6:24 AM Nevo Sagi via R-sig-meta-analysis <
> r-sig-meta-analysis using r-project.org> wrote:
>
>> Dear list members, I have a follow-up question.
>>
>> In my dataset I have about 500 experiments (i.e., observations) across 50
>> articles (i.e., references), but the moderators in question change across
>> observations only within 7 of the references. Consequently, my rma.mv
>> model
>> that uses ~1|Reference as a random effect is over-sensitive to the data
>> from these 7 studies compared to the others.
>> In such a case, if I use a rma.mv (or rma.uni) model without a random
>> effect, would it be more reliable?
>> And if I do use such a model, how do I compute the R^2 for each moderator
>> (as sigma^2 is inapplicable)?
>>
>> Thanks again,
>> Nevo Sagi
>>
>> On Mon, Jun 5, 2023 at 10:52 AM Nevo Sagi <nevosagi8 using gmail.com> wrote:
>>
>> > Dear Wolgang,
>> >
>> > Thank you for your feedback.
>> >
>> > It turns out that I misplaced the equation terms when calculating the
>> > pseudo-R^2.
>> >
>> > All the best,
>> > Nevo
>> >
>> > On Thu, Jun 1, 2023 at 3:30 PM Viechtbauer, Wolfgang (NP) <
>> > wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>> >
>> >> Dear Nevo,
>> >>
>> >> Please see my responses below.
>> >>
>> >> Best,
>> >> Wolfgang
>> >>
>> >> >-----Original Message-----
>> >> >From: R-sig-meta-analysis [mailto:
>> >> r-sig-meta-analysis-bounces using r-project.org] On
>> >> >Behalf Of Nevo Sagi via R-sig-meta-analysis
>> >> >Sent: Thursday, 04 May, 2023 11:09
>> >> >To: r-sig-meta-analysis using r-project.org
>> >> >Cc: Nevo Sagi
>> >> >Subject: [R-meta] Questions regarding REML and FE models and R^2
>> >> calculation in
>> >> >metafor
>> >> >
>> >> >Dear list members,
>> >> >
>> >> >I conducted a meta-analysis on the role of climate in mediating a
>> >> specific
>> >> >ecological process, using the *metafor *package in R.
>> >> >This is actually a meta-regression, using the rma.mv function, with
>> >> >*temperature *and *precipitation *as moderators, along with some other
>> >> >moderators related to experimental design. I also use reference as a
>> >> random
>> >> >effect ('random = ~1|*Reference'*), as some references include more
>> than
>> >> >one experiment.
>> >> >
>> >> >*1. FE vs REML model:*
>> >> >After reading Wolfgang Viechtbauer's blog post
>> >> ><https://wviechtb.github.io/metafor/reference/misc-models.html> on
>> the
>> >> >differences between fixed-effects and random-effects models in the
>> >> >*metafor *package, I decided to use the FE method, because the
>> studies I
>> >> >gathered are not a random sample of the population of hypothetical
>> >> studies.
>> >> >Instead, the sample is biased by underrepresentation of some climates
>> and
>> >> >overrepresentation of others.
>> >> >I wonder whether my interpretation of the difference between FE and
>> REML
>> >> >models is correct, and would like to get some feedback on it.
>> >>
>> >> I don't think this is really a good reason for using a FE model,
>> because
>> >> the underrepresentation of some climates and overrepresentation of
>> others
>> >> will affect your results either way. The bigger question is if climate
>> is
>> >> an important moderator, which you can examine via meta-regression.
>> >>
>> >> >*2. R^2 calculation:*
>> >> >Reviewers of my manuscript required that I provide R-squared values
>> for
>> >> >each of the climate moderators.
>> >> >Using the *metafor *package, only rma.uni models (where random
>> variables
>> >> >cannot be specified) provide R^2 estimation.
>> >> >In a previous conversation in this mailing list, Wolfgang indicated
>> that
>> >> >pseudo-R^2 can be calculated based on the variance (sigma2) reported
>> by
>> >> >models including and excluding the moderator in question:
>> >> >*(res0$sigma2 - res1$sigma2) / res0$sigma2*
>> >> >*where 'res0' is the model without coefficients and 'res1' the model
>> >> with.*
>> >> >
>> >> >I have two problems with this solution:
>> >> >1. FE models do not provide variance components (sigma2). Therefore,
>> the
>> >> >pseudo R-squared can be calculated only for REML models. I guess this
>> can
>> >> >be explained by the nature of the models, which I don't fully
>> understand.
>> >>
>> >> Yes, this approach to calculating such pseudo-R^2 values only works in
>> RE
>> >> models.
>> >>
>> >> >2. When using REML models and performing the above calculation, I get
>> >> weird
>> >> >results. For example, one of the pseudo R^2 values was above 1. This
>> >> cannot
>> >> >mean that the moderator explained more than 100% of the variance in
>> the
>> >> >effect size. How comparable is this pseudo R^2 for the standard R^2 of
>> >> >simpler models?
>> >>
>> >> This is mathematically impossible. (res0$sigma2 - res1$sigma2) /
>> >> res0$sigma2 is the same as 1 - res1$sigma2 / res0$sigma2 and the second
>> >> term cannot be negative, so the resulting value cannot be larger than
>> 1.
>> >>
>> >> >To conclude, I will be glad to get feedback on both problems:
>> >> >1. Should I use a random-effect or fixed-effect model?
>> >> >2. How do I get a reliable R^2 or an alternative measure of goodness
>> of
>> >> fit
>> >> >for single-moderator models that include a random structure and a
>> >> sampling
>> >> >variance?
>> >> >
>> >> >Thank you very much,
>> >> >
>> >> >Nevo Sagi
>> >> >
>> >> >--
>> >> >Dr. Nevo Sagi
>> >> >
>> >> >Prof. Dror Hawlena's Risk-Management Ecology Lab
>> >> >Department of Ecology, Evolution & Behavior
>> >> >The Alexander Silberman Institute of Life Sciences
>> >> >The Hebrew University of Jerusalem
>> >> >Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel.
>> >>
>> >
>> >
>> > --
>> > Dr. Nevo Sagi
>> >
>> > Prof. Dror Hawlena's Risk-Management Ecology Lab
>> > Department of Ecology, Evolution & Behavior
>> > The Alexander Silberman Institute of Life Sciences
>> > The Hebrew University of Jerusalem
>> > Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel.
>> >
>>
>>
>> --
>> Dr. Nevo Sagi
>>
>> Prof. Dror Hawlena's Risk-Management Ecology Lab
>> Department of Ecology, Evolution & Behavior
>> The Alexander Silberman Institute of Life Sciences
>> The Hebrew University of Jerusalem
>> Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel.
>>
>>         [[alternative HTML version deleted]]
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
>> To manage your subscription to this mailing list, go to:
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>
>

-- 
Dr. Nevo Sagi

Prof. Dror Hawlena's Risk-Management Ecology Lab
Department of Ecology, Evolution & Behavior
The Alexander Silberman Institute of Life Sciences
The Hebrew University of Jerusalem
Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel.

	[[alternative HTML version deleted]]