[R-meta] Questions regarding REML and FE models and R^2 calculation in metafor

[Nevo Sagi]

Dear list members, I have a follow-up question.

In my dataset I have about 500 experiments (i.e., observations) across 50
articles (i.e., references), but the moderators in question change across
observations only within 7 of the references. Consequently, my rma.mv model
that uses ~1|Reference as a random effect is over-sensitive to the data
from these 7 studies compared to the others.
In such a case, if I use a rma.mv (or rma.uni) model without a random
effect, would it be more reliable?
And if I do use such a model, how do I compute the R^2 for each moderator
(as sigma^2 is inapplicable)?

Thanks again,
Nevo Sagi

On Mon, Jun 5, 2023 at 10:52 AM Nevo Sagi <nevosagi8 using gmail.com> wrote:

> Dear Wolgang,
>
> Thank you for your feedback.
>
> It turns out that I misplaced the equation terms when calculating the
> pseudo-R^2.
>
> All the best,
> Nevo
>
> On Thu, Jun 1, 2023 at 3:30 PM Viechtbauer, Wolfgang (NP) <
> wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>
>> Dear Nevo,
>>
>> Please see my responses below.
>>
>> Best,
>> Wolfgang
>>
>> >-----Original Message-----
>> >From: R-sig-meta-analysis [mailto:
>> r-sig-meta-analysis-bounces using r-project.org] On
>> >Behalf Of Nevo Sagi via R-sig-meta-analysis
>> >Sent: Thursday, 04 May, 2023 11:09
>> >To: r-sig-meta-analysis using r-project.org
>> >Cc: Nevo Sagi
>> >Subject: [R-meta] Questions regarding REML and FE models and R^2
>> calculation in
>> >metafor
>> >
>> >Dear list members,
>> >
>> >I conducted a meta-analysis on the role of climate in mediating a
>> specific
>> >ecological process, using the *metafor *package in R.
>> >This is actually a meta-regression, using the rma.mv function, with
>> >*temperature *and *precipitation *as moderators, along with some other
>> >moderators related to experimental design. I also use reference as a
>> random
>> >effect ('random = ~1|*Reference'*), as some references include more than
>> >one experiment.
>> >
>> >*1. FE vs REML model:*
>> >After reading Wolfgang Viechtbauer's blog post
>> ><https://wviechtb.github.io/metafor/reference/misc-models.html> on the
>> >differences between fixed-effects and random-effects models in the
>> >*metafor *package, I decided to use the FE method, because the studies I
>> >gathered are not a random sample of the population of hypothetical
>> studies.
>> >Instead, the sample is biased by underrepresentation of some climates and
>> >overrepresentation of others.
>> >I wonder whether my interpretation of the difference between FE and REML
>> >models is correct, and would like to get some feedback on it.
>>
>> I don't think this is really a good reason for using a FE model, because
>> the underrepresentation of some climates and overrepresentation of others
>> will affect your results either way. The bigger question is if climate is
>> an important moderator, which you can examine via meta-regression.
>>
>> >*2. R^2 calculation:*
>> >Reviewers of my manuscript required that I provide R-squared values for
>> >each of the climate moderators.
>> >Using the *metafor *package, only rma.uni models (where random variables
>> >cannot be specified) provide R^2 estimation.
>> >In a previous conversation in this mailing list, Wolfgang indicated that
>> >pseudo-R^2 can be calculated based on the variance (sigma2) reported by
>> >models including and excluding the moderator in question:
>> >*(res0$sigma2 - res1$sigma2) / res0$sigma2*
>> >*where 'res0' is the model without coefficients and 'res1' the model
>> with.*
>> >
>> >I have two problems with this solution:
>> >1. FE models do not provide variance components (sigma2). Therefore, the
>> >pseudo R-squared can be calculated only for REML models. I guess this can
>> >be explained by the nature of the models, which I don't fully understand.
>>
>> Yes, this approach to calculating such pseudo-R^2 values only works in RE
>> models.
>>
>> >2. When using REML models and performing the above calculation, I get
>> weird
>> >results. For example, one of the pseudo R^2 values was above 1. This
>> cannot
>> >mean that the moderator explained more than 100% of the variance in the
>> >effect size. How comparable is this pseudo R^2 for the standard R^2 of
>> >simpler models?
>>
>> This is mathematically impossible. (res0$sigma2 - res1$sigma2) /
>> res0$sigma2 is the same as 1 - res1$sigma2 / res0$sigma2 and the second
>> term cannot be negative, so the resulting value cannot be larger than 1.
>>
>> >To conclude, I will be glad to get feedback on both problems:
>> >1. Should I use a random-effect or fixed-effect model?
>> >2. How do I get a reliable R^2 or an alternative measure of goodness of
>> fit
>> >for single-moderator models that include a random structure and a
>> sampling
>> >variance?
>> >
>> >Thank you very much,
>> >
>> >Nevo Sagi
>> >
>> >--
>> >Dr. Nevo Sagi
>> >
>> >Prof. Dror Hawlena's Risk-Management Ecology Lab
>> >Department of Ecology, Evolution & Behavior
>> >The Alexander Silberman Institute of Life Sciences
>> >The Hebrew University of Jerusalem
>> >Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel.
>>
>
>
> --
> Dr. Nevo Sagi
>
> Prof. Dror Hawlena's Risk-Management Ecology Lab
> Department of Ecology, Evolution & Behavior
> The Alexander Silberman Institute of Life Sciences
> The Hebrew University of Jerusalem
> Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel.
>


-- 
Dr. Nevo Sagi

Prof. Dror Hawlena's Risk-Management Ecology Lab
Department of Ecology, Evolution & Behavior
The Alexander Silberman Institute of Life Sciences
The Hebrew University of Jerusalem
Edmond J. Safra Campus at Givat Ram, Jerusalem 9190401, Israel.

	[[alternative HTML version deleted]]