[R-meta] R-square (change) as effect size

[Viechtbauer, Wolfgang (NP)]

Hi Paul,

What kind of data do you actually have? Do you have the two adjusted R^2 values for the two models of interest for each study? Or do you have studies where some provide the adjusted R^2 for the first model and other studies that provide the adjusted R^2 for the other model?

> Would it be possible to transform the adjusted R-square to make its distribution
> more normal, take its squareroot and then treat it as a correlation coefficient
> using rma()?

No, it's not that simple. Just to give a counter-example: Say you have a simple regression model of the form y = beta0 + beta1 x + e and take the R^2 value from that model. Then sqrt(R^2) is equal to the *absolute value* of the correlation between x and y. Since it is an absolute value, it isn't going to behave like a regular/signed correlation coefficient and cannot be treated as such.

Best,
Wolfgang

>-----Original Message-----
>From: Hanel, Paul H P [mailto:p.hanel using essex.ac.uk]
>Sent: Tuesday, 18 April, 2023 12:00
>To: R Special Interest Group for Meta-Analysis; Michael Dewey
>Cc: Viechtbauer, Wolfgang (NP)
>Subject: RE: [R-meta] R-square (change) as effect size
>
>Dear Wolfgang and Michael,
>
>Thank you. After also having had a look at the links you provided, I am not sure
>whether it would be best to focus on the adjusted R-square change or the adjusted
>R-squares. Focusing on the adjusted R-square values seems to make slightly more
>sense to me, but either should work for what I have in mind. Would it be possible
>to transform the adjusted R-square to make its distribution more normal, take its
>squareroot and then treat it as a correlation coefficient using rma()?
>
>Regarding the third point, the 'directionless' of R-square: This is not overly
>relevant to my research question. I am interested in whether a set of five
>predictors (personality traits, Big-5) explain more or less variance in a range
>of outcome variable than a set of ten predictors (human values). If a specific
>personality trait or value is correlated with the outcome variables is something
>for future research - or has already been done.
>
>Best,
>Paul
>
>-----Original Message-----
>From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf
>Of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
>Sent: 18 April 2023 09:26
>To: Michael Dewey <lists using dewey.myzen.co.uk>; R Special Interest Group for Meta-
>Analysis <r-sig-meta-analysis using r-project.org>
>Cc: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
>Subject: Re: [R-meta] R-square (change) as effect size
>
>Great, thanks. Also found this thread:
>
>https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2021-March/002708.html
>
>which goes a bit in the same direction.
>
>If one focuses on the difference between the two R^2 values, then this might hold
>some promise, but there are still these pesky little technical details to figure
>out -- (approximate) normality of the sampling distribution and the sampling
>variance of such a difference.
>
>P.S.: Just as a reminder to all, there is a custom Google search set up for
>searching the mailing list archives here:
>https://cse.google.com/cse?cx=ee4b2e6c93b6a9667 One caveat: Google crawls the
>archives only periodically, so recent posts will not show up in the search (for
>example, I tried a search for "R2" and it doesn't bring up the thread from
>January, but it did lead me to the one from 2021).
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: Michael Dewey [mailto:lists using dewey.myzen.co.uk]
>>Sent: Monday, 17 April, 2023 17:04
>>To: R Special Interest Group for Meta-Analysis
>>Cc: Viechtbauer, Wolfgang (NP)
>>Subject: Re: [R-meta] R-square (change) as effect size
>>
>>Dear Wolfgang, you were looking for
>>https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2023-January/004331.
>>html and the surrounding thread although my hint there only addressed
>>you point 3.
>>
>>Michael
>>
>>On 17/04/2023 13:23, Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
>>wrote:
>>> Dear Paul,
>>>
>>> I think the issue of using R^2 as an effect size measure for a
>>> meta-analysis
>>has come up before on this mailing list, although I can't find the
>>threads right now. In any case, there are several practical issues:
>>>
>>> 1) The sampling distribution of (adjusted) R^2 is not normal, so one
>>> needs to
>>figure out some appropriate normalizing transformation.
>>>
>>> 2) One also needs to figure out the sampling variance of the
>>> (transformed)
>>(adjusted) R^2 value.
>>>
>>> 3) One can also debate the usefulness of meta-analyzing a 'directionless'
>>measure such as (adjusted) R^2. If you focus on the difference in
>>(adjusted) R^2 though (of two non-nested models), then I think this
>>issue is less concerning (and the sampling distribution of such a
>>difference might actually be somewhat normal). However, this then
>>raises another problem: The two (transformed)
>>(adjusted) R^2 values are not independent if they come from the same
>>sample and so one now also needs to figure out their covariance. If
>>they do not come from the same sample, then this alleviates this
>>particular issue, but makes the evidence much weaker due to potential
>confounding.
>>>
>>> Best,
>>> Wolfgang