[R-meta] R-square (change) as effect size

[Viechtbauer, Wolfgang (NP)]

Sidenote: Posting in plain-text will prevent tables from getting scrambled up.

If you have the raw data, then this of course opens up many possibilities, especially for obtaining the sampling variances (and covariances) of whatever values you are interested in. In particular, one could use bootstrapping to obtain these values.

In particular, for each study, you could compute y_1 = (Rt2 - Rv2) for religiosity and y_2 = (Rt2 - Rv2) for well-being in each bootstrap sample and then use var(y_1) and var(y_2) from these bootstrap replicates as the sampling variance of the two estimates and cov(y_1, y_2) as their covariance. It would also be good to examine the bootstrap distributions of y_1 and y_2 to see if they are somewhat normal.

Then you can construct a dataset like this:

study estimate dv
1     y_1      religiosity
1     y_2      well-being

with the corresponding V matrix containing the 2x2 blocks with var(y_1), cov(y_1, y_2), and var(y_2).

And now you are in the exact same situation as this:

https://wviechtb.github.io/metadat/reference/dat.berkey1998.html

so you can use

res <- rma.mv(estimate, V, mods = ~ dv, random = ~ dv | study, struct="UN", data=dat)

to estimate the average difference between Rt2 and Rv2 for each DV and test if there is a difference between thenm.

Best,
Wolfgang

>-----Original Message-----
>From: Hanel, Paul H P [mailto:p.hanel using essex.ac.uk]
>Sent: Wednesday, 19 April, 2023 21:11
>To: R Special Interest Group for Meta-Analysis; Viechtbauer, Wolfgang (NP)
>Subject: RE: [R-meta] R-square (change) as effect size
>
>PS: Just realised the table doesn't look like it should be. So I copied in a
>screenshot of it and uploaded it to a Google Doc, just in case
>
>https://docs.google.com/document/d/1xICNhHQ_Te_riYjozG90d6idqvc8-
>PSSk7eLZkzG2ls/edit?usp=sharing
>
>-----Original Message-----
>From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On Behalf
>Of Hanel, Paul H P via R-sig-meta-analysis
>Sent: 19 April 2023 20:04
>To: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>; R
>Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-project.org>
>Cc: Hanel, Paul H P <p.hanel using essex.ac.uk>
>Subject: Re: [R-meta] R-square (change) as effect size
>
>Hi Wolfgang,
>
>I have the adjusted R-square and adjusted R-square change values for both sets of
>predictors and each DV. Luckily, many researchers were forthcoming and shared
>their raw data, since only a few papers reported the required hierarchical
>regression results.
>
>See below for an example table (Moderators are omitted). It looks like
>personality traits (Big-5) explain more variance in well-being than human values,
>whereas values explain more variance in religiosity than traits.
>
>Thank you,
>
>Paul
>
>Paper ID Study ID DV          N     Rt2   Rt2 (change)   Rv2   Rv2 (change)
>1        1        Religiosity 987   .05   .02            .15   .12
>1        2        Well-being  789   .20   .15            .10   .05
>2        1        Religiosity 654   .05   .02            .15   .12
>2        2        Well-being  456   .30   .25            .10   .05
>
>Note. Rt2: Amount of variance traits explain in DV, Rt2 (change): Amount of
>variance traits explain beyond values in DV; Rv2: Amount of variance values
>explain in DV, Rv2 (change): Amount of variance values explain beyond values in
>DV. All adjusted R2-values. Rt2 + Rv2 (change) =~ Rv2 + Rt2 (change).
>
>-----Original Message-----
>From: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
>Sent: 18 April 2023 11:17
>To: Hanel, Paul H P <p.hanel using essex.ac.uk>; R Special Interest Group for Meta-
>Analysis <r-sig-meta-analysis using r-project.org>
>Subject: RE: [R-meta] R-square (change) as effect size
>
>Hi Paul,
>
>What kind of data do you actually have? Do you have the two adjusted R^2 values
>for the two models of interest for each study? Or do you have studies where some
>provide the adjusted R^2 for the first model and other studies that provide the
>adjusted R^2 for the other model?
>
>> Would it be possible to transform the adjusted R-square to make its
>> distribution more normal, take its squareroot and then treat it as a
>> correlation coefficient using rma()?
>
>No, it's not that simple. Just to give a counter-example: Say you have a simple
>regression model of the form y = beta0 + beta1 x + e and take the R^2 value from
>that model. Then sqrt(R^2) is equal to the *absolute value* of the correlation
>between x and y. Since it is an absolute value, it isn't going to behave like a
>regular/signed correlation coefficient and cannot be treated as such.
>
>Best,
>Wolfgang