[R-meta] R-square (change) as effect size

Wed Apr 19 21:10:34 CEST 2023

PS: Just realised the table doesn't look like it should be. So I copied in a screenshot of it and uploaded it to a Google Doc, just in case https://docs.google.com/document/d/1xICNhHQ_Te_riYjozG90d6idqvc8-PSSk7eLZkzG2ls/edit?usp=sharing

[cid:image001.png using 01D972FA.B3D9B4C0]

-----Original Message-----
From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf Of Hanel, Paul H P via R-sig-meta-analysis
Sent: 19 April 2023 20:04
To: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>; R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-project.org>
Cc: Hanel, Paul H P <p.hanel using essex.ac.uk>
Subject: Re: [R-meta] R-square (change) as effect size

Hi Wolfgang,

I have the adjusted R-square and adjusted R-square change values for both sets of predictors and each DV. Luckily, many researchers were forthcoming and shared their raw data, since only a few papers reported the required hierarchical regression results.

See below for an example table (Moderators are omitted). It looks like personality traits (Big-5) explain more variance in well-being than human values, whereas values explain more variance in religiosity than traits.

Thank you,

Paul

Paper ID

Study ID

DV

N

Rt2

Rt2 (change)

Rv2

Rv2 (change)

1

1

Religiosity

987

.05

.02

.15

.12

1

2

Well-being

789

.20

.15

.10

.05

2

1

Religiosity

654

.05

.02

.15

.12

2

2

Well-being

456

.30

.25

.10

.05

Note. Rt2: Amount of variance traits explain in DV, Rt2 (change): Amount of variance traits explain beyond values in DV; Rv2: Amount of variance values explain in DV, Rv2 (change): Amount of variance values explain beyond values in DV. All adjusted R2-values. Rt2 + Rv2 (change) � Rv2 + Rt2 (change).

-----Original Message-----

From: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl<mailto:wolfgang.viechtbauer using maastrichtuniversity.nl>>

Sent: 18 April 2023 11:17

To: Hanel, Paul H P <p.hanel using essex.ac.uk<mailto:p.hanel using essex.ac.uk>>; R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org>>

Subject: RE: [R-meta] R-square (change) as effect size

CAUTION: This email was sent from outside the University of Essex. Please do not click any links or open any attachments unless you recognise and trust the sender. If you are unsure whether the content of the email is safe or have any other queries, please contact the IT Helpdesk.

Hi Paul,

What kind of data do you actually have? Do you have the two adjusted R^2 values for the two models of interest for each study? Or do you have studies where some provide the adjusted R^2 for the first model and other studies that provide the adjusted R^2 for the other model?

> Would it be possible to transform the adjusted R-square to make its

> distribution more normal, take its squareroot and then treat it as a

> correlation coefficient using rma()?

No, it's not that simple. Just to give a counter-example: Say you have a simple regression model of the form y = beta0 + beta1 x + e and take the R^2 value from that model. Then sqrt(R^2) is equal to the *absolute value* of the correlation between x and y. Since it is an absolute value, it isn't going to behave like a regular/signed correlation coefficient and cannot be treated as such.

Best,

Wolfgang

>-----Original Message-----

>From: Hanel, Paul H P [mailto:p.hanel using essex.ac.uk]

>Sent: Tuesday, 18 April, 2023 12:00

>To: R Special Interest Group for Meta-Analysis; Michael Dewey

>Cc: Viechtbauer, Wolfgang (NP)

>Subject: RE: [R-meta] R-square (change) as effect size

>

>Dear Wolfgang and Michael,

>

>Thank you. After also having had a look at the links you provided, I am

>not sure whether it would be best to focus on the adjusted R-square

>change or the adjusted R-squares. Focusing on the adjusted R-square

>values seems to make slightly more sense to me, but either should work

>for what I have in mind. Would it be possible to transform the adjusted

>R-square to make its distribution more normal, take its squareroot and then treat it as a correlation coefficient using rma()?

>

>Regarding the third point, the 'directionless' of R-square: This is not

>overly relevant to my research question. I am interested in whether a

>set of five predictors (personality traits, Big-5) explain more or less

>variance in a range of outcome variable than a set of ten predictors

>(human values). If a specific personality trait or value is correlated

>with the outcome variables is something for future research - or has already been done.

>

>Best,

>Paul

>

>-----Original Message-----

>From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org<mailto:r-sig-meta-analysis-bounces using r-project.org<mailto:r-sig-meta-analysis-bounces using r-project.org%3cmailto:r-sig-meta-analysis-bounces using r-project.org>>>

>On Behalf Of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis

>Sent: 18 April 2023 09:26

>To: Michael Dewey <lists using dewey.myzen.co.uk<mailto:lists using dewey.myzen.co.uk<mailto:lists using dewey.myzen.co.uk%3cmailto:lists using dewey.myzen.co.uk>>>; R Special Interest Group

>for Meta- Analysis <r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org<mailto:r-sig-meta-analysis using r-project.org%3cmailto:r-sig-meta-analysis using r-project.org>>>

>Cc: Viechtbauer, Wolfgang (NP)

><wolfgang.viechtbauer using maastrichtuniversity.nl<mailto:wolfgang.viechtbauer using maastrichtuniversity.nl<mailto:wolfgang.viechtbauer using maastrichtuniversity.nl%3cmailto:wolfgang.viechtbauer using maastrichtuniversity.nl>>>

>Subject: Re: [R-meta] R-square (change) as effect size

>

>Great, thanks. Also found this thread:

>

>https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2021-March/002708.ht

>ml

>

>which goes a bit in the same direction.

>

>If one focuses on the difference between the two R^2 values, then this

>might hold some promise, but there are still these pesky little

>technical details to figure out -- (approximate) normality of the

>sampling distribution and the sampling variance of such a difference.

>

>P.S.: Just as a reminder to all, there is a custom Google search set up

>for searching the mailing list archives here:

>https://cse.google.com/cse?cx=ee4b2e6c93b6a9667 One caveat: Google

>crawls the archives only periodically, so recent posts will not show up

>in the search (for example, I tried a search for "R2" and it doesn't

>bring up the thread from January, but it did lead me to the one from 2021).

>

>Best,

>Wolfgang

>

>>-----Original Message-----

>>From: Michael Dewey [mailto:lists using dewey.myzen.co.uk]

>>Sent: Monday, 17 April, 2023 17:04

>>To: R Special Interest Group for Meta-Analysis

>>Cc: Viechtbauer, Wolfgang (NP)

>>Subject: Re: [R-meta] R-square (change) as effect size

>>

>>Dear Wolfgang, you were looking for

>>https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2023-January/004331.

>>html and the surrounding thread although my hint there only addressed

>>you point 3.

>>

>>Michael

>>

>>On 17/04/2023 13:23, Viechtbauer, Wolfgang (NP) via

>>R-sig-meta-analysis

>>wrote:

>>> Dear Paul,

>>>

>>> I think the issue of using R^2 as an effect size measure for a

>>> meta-analysis

>>has come up before on this mailing list, although I can't find the

>>threads right now. In any case, there are several practical issues:

>>>

>>> 1) The sampling distribution of (adjusted) R^2 is not normal, so one

>>> needs to

>>figure out some appropriate normalizing transformation.

>>>

>>> 2) One also needs to figure out the sampling variance of the

>>> (transformed)

>>(adjusted) R^2 value.

>>>

>>> 3) One can also debate the usefulness of meta-analyzing a 'directionless'

>>measure such as (adjusted) R^2. If you focus on the difference in

>>(adjusted) R^2 though (of two non-nested models), then I think this

>>issue is less concerning (and the sampling distribution of such a

>>difference might actually be somewhat normal). However, this then

>>raises another problem: The two (transformed)

>>(adjusted) R^2 values are not independent if they come from the same

>>sample and so one now also needs to figure out their covariance. If

>>they do not come from the same sample, then this alleviates this

>>particular issue, but makes the evidence much weaker due to potential

>confounding.

>>>

>>> Best,

>>> Wolfgang

               [[alternative HTML version deleted]]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20230419/846d9fea/attachment-0001.html>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 23911 bytes
Desc: image001.png
URL: <https://stat.ethz.ch/pipermail/r-sig-meta-analysis/attachments/20230419/846d9fea/attachment-0001.png>