[R-meta] Questions regarding REML and FE models and R^2 calculation in metafor

[Nevo Sagi]

Dear Wolgang,

Thank you for your feedback.

It turns out that I misplaced the equation terms when calculating the
pseudo-R^2.

All the best,
Nevo

On Thu, Jun 1, 2023 at 3:30 PM Viechtbauer, Wolfgang (NP) <
wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:

> Dear Nevo,
>
> Please see my responses below.
>
> Best,
> Wolfgang
>
> >-----Original Message-----
> >From: R-sig-meta-analysis [mailto:
> r-sig-meta-analysis-bounces using r-project.org] On
> >Behalf Of Nevo Sagi via R-sig-meta-analysis
> >Sent: Thursday, 04 May, 2023 11:09
> >To: r-sig-meta-analysis using r-project.org
> >Cc: Nevo Sagi
> >Subject: [R-meta] Questions regarding REML and FE models and R^2
> calculation in
> >metafor
> >
> >Dear list members,
> >
> >I conducted a meta-analysis on the role of climate in mediating a specific
> >ecological process, using the *metafor *package in R.
> >This is actually a meta-regression, using the rma.mv function, with
> >*temperature *and *precipitation *as moderators, along with some other
> >moderators related to experimental design. I also use reference as a
> random
> >effect ('random = ~1|*Reference'*), as some references include more than
> >one experiment.
> >
> >*1. FE vs REML model:*
> >After reading Wolfgang Viechtbauer's blog post
> ><https://wviechtb.github.io/metafor/reference/misc-models.html> on the
> >differences between fixed-effects and random-effects models in the
> >*metafor *package, I decided to use the FE method, because the studies I
> >gathered are not a random sample of the population of hypothetical
> studies.
> >Instead, the sample is biased by underrepresentation of some climates and
> >overrepresentation of others.
> >I wonder whether my interpretation of the difference between FE and REML
> >models is correct, and would like to get some feedback on it.
>
> I don't think this is really a good reason for using a FE model, because
> the underrepresentation of some climates and overrepresentation of others
> will affect your results either way. The bigger question is if climate is
> an important moderator, which you can examine via meta-regression.
>
> >*2. R^2 calculation:*
> >Reviewers of my manuscript required that I provide R-squared values for
> >each of the climate moderators.
> >Using the *metafor *package, only rma.uni models (where random variables
> >cannot be specified) provide R^2 estimation.
> >In a previous conversation in this mailing list, Wolfgang indicated that
> >pseudo-R^2 can be calculated based on the variance (sigma2) reported by
> >models including and excluding the moderator in question:
> >*(res0$sigma2 - res1$sigma2) / res0$sigma2*
> >*where 'res0' is the model without coefficients and 'res1' the model
> with.*
> >
> >I have two problems with this solution:
> >1. FE models do not provide variance components (sigma2). Therefore, the
> >pseudo R-squared can be calculated only for REML models. I guess this can
> >be explained by the nature of the models, which I don't fully understand.
>
> Yes, this approach to calculating such pseudo-R^2 values only works in RE
> models.
>
> >2. When using REML models and performing the above calculation, I get
> weird
> >results. For example, one of the pseudo R^2 values was above 1. This
> cannot
> >mean that the moderator explained more than 100% of the variance in the
> >effect size. How comparable is this pseudo R^2 for the standard R^2 of
> >simpler models?
>
> This is mathematically impossible. (res0$sigma2 - res1$sigma2) /
> res0$sigma2 is the same as 1 - res1$sigma2 / res0$sigma2 and the second
> term cannot be negative, so the resulting value cannot be larger than 1.
>
> >To conclude, I will be glad to get feedback on both problems:
> >1. Should I use a random-effect or fixed-effect model?
> >2. How do I get a reliable R^2 or an alternative measure of goodness of
> fit
> >for single-moderator models that include a random structure and a sampling
> >variance?
> >
> >Thank you very much,
> >
> >Nevo Sagi
> >
> >--
> >Dr. Nevo Sagi
> >
> >Prof. Dror Hawlena's Risk-Management Ecology Lab
> >Department of Ecology, Evolution & Behavior
> >The Alexander Silberman Institute of Life Sciences
> >The Hebrew University of Jerusalem
> >Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel.
>


-- 
Dr. Nevo Sagi

Prof. Dror Hawlena's Risk-Management Ecology Lab
Department of Ecology, Evolution & Behavior
The Alexander Silberman Institute of Life Sciences
The Hebrew University of Jerusalem
Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel.

	[[alternative HTML version deleted]]