[R-meta] R-square (change) as effect size

[Hanel, Paul H P]

Hi Wolfgang,

Thank you. 

Sorry, I should have been clearer. I have the raw data for perhaps half of the studies. 

I take that a meta-analysis would only be possible for the studies for which I have the raw data using the method outlined below, but not for the studies for which I only have the adjusted R-squares and adjusted R-square changes?

Best wishes,
Paul

-----Original Message-----
From: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl> 
Sent: 20 April 2023 17:01
To: Hanel, Paul H P <p.hanel using essex.ac.uk>; R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-project.org>
Subject: RE: [R-meta] R-square (change) as effect size

CAUTION: This email was sent from outside the University of Essex. Please do not click any links or open any attachments unless you recognise and trust the sender. If you are unsure whether the content of the email is safe or have any other queries, please contact the IT Helpdesk.

Sidenote: Posting in plain-text will prevent tables from getting scrambled up.

If you have the raw data, then this of course opens up many possibilities, especially for obtaining the sampling variances (and covariances) of whatever values you are interested in. In particular, one could use bootstrapping to obtain these values.

In particular, for each study, you could compute y_1 = (Rt2 - Rv2) for religiosity and y_2 = (Rt2 - Rv2) for well-being in each bootstrap sample and then use var(y_1) and var(y_2) from these bootstrap replicates as the sampling variance of the two estimates and cov(y_1, y_2) as their covariance. It would also be good to examine the bootstrap distributions of y_1 and y_2 to see if they are somewhat normal.

Then you can construct a dataset like this:

study estimate dv
1     y_1      religiosity
1     y_2      well-being

with the corresponding V matrix containing the 2x2 blocks with var(y_1), cov(y_1, y_2), and var(y_2).

And now you are in the exact same situation as this:

https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwviechtb.github.io%2fmetadat%2freference%2fdat.berkey1998.html&c=E,1,w6DcQI3Tp2q5TItSiACCatYNaeLukdqL9SBmZb879IY_lWR8NDfjK5dbJA3lFfU87sRlW-RIVr97ctb7JBu0hgn3oip1adl5nY6vYdEHmuHdRd_rsjEIQDbl&typo=1

so you can use

res <- https://linkprotect.cudasvc.com/url?a=https%3a%2f%2frma.mv&c=E,1,FxhB1OcOXdnxSGLwlT5Twnc73lbSgCIJF4GWiKLOOhov8809880vgxZXr7yyePTUsFMkm72qgpHZHMuyEcqJkJZsZ3YG7Bu17qHplkaxlBmt&typo=1(estimate, V, mods = ~ dv, random = ~ dv | study, struct="UN", data=dat)

to estimate the average difference between Rt2 and Rv2 for each DV and test if there is a difference between thenm.

Best,
Wolfgang

>-----Original Message-----
>From: Hanel, Paul H P [mailto:p.hanel using essex.ac.uk]
>Sent: Wednesday, 19 April, 2023 21:11
>To: R Special Interest Group for Meta-Analysis; Viechtbauer, Wolfgang 
>(NP)
>Subject: RE: [R-meta] R-square (change) as effect size
>
>PS: Just realised the table doesn't look like it should be. So I copied 
>in a screenshot of it and uploaded it to a Google Doc, just in case
>
>https://docs.google.com/document/d/1xICNhHQ_Te_riYjozG90d6idqvc8-
>PSSk7eLZkzG2ls/edit?usp=sharing
>
>-----Original Message-----
>From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> 
>On Behalf Of Hanel, Paul H P via R-sig-meta-analysis
>Sent: 19 April 2023 20:04
>To: Viechtbauer, Wolfgang (NP) 
><wolfgang.viechtbauer using maastrichtuniversity.nl>; R Special Interest 
>Group for Meta-Analysis <r-sig-meta-analysis using r-project.org>
>Cc: Hanel, Paul H P <p.hanel using essex.ac.uk>
>Subject: Re: [R-meta] R-square (change) as effect size
>
>Hi Wolfgang,
>
>I have the adjusted R-square and adjusted R-square change values for 
>both sets of predictors and each DV. Luckily, many researchers were 
>forthcoming and shared their raw data, since only a few papers reported 
>the required hierarchical regression results.
>
>See below for an example table (Moderators are omitted). It looks like 
>personality traits (Big-5) explain more variance in well-being than 
>human values, whereas values explain more variance in religiosity than traits.
>
>Thank you,
>
>Paul
>
>Paper ID Study ID DV          N     Rt2   Rt2 (change)   Rv2   Rv2 (change)
>1        1        Religiosity 987   .05   .02            .15   .12
>1        2        Well-being  789   .20   .15            .10   .05
>2        1        Religiosity 654   .05   .02            .15   .12
>2        2        Well-being  456   .30   .25            .10   .05
>
>Note. Rt2: Amount of variance traits explain in DV, Rt2 (change): 
>Amount of variance traits explain beyond values in DV; Rv2: Amount of 
>variance values explain in DV, Rv2 (change): Amount of variance values 
>explain beyond values in DV. All adjusted R2-values. Rt2 + Rv2 (change) =~ Rv2 + Rt2 (change).
>
>-----Original Message-----
>From: Viechtbauer, Wolfgang (NP) 
><wolfgang.viechtbauer using maastrichtuniversity.nl>
>Sent: 18 April 2023 11:17
>To: Hanel, Paul H P <p.hanel using essex.ac.uk>; R Special Interest Group for 
>Meta- Analysis <r-sig-meta-analysis using r-project.org>
>Subject: RE: [R-meta] R-square (change) as effect size
>
>Hi Paul,
>
>What kind of data do you actually have? Do you have the two adjusted 
>R^2 values for the two models of interest for each study? Or do you 
>have studies where some provide the adjusted R^2 for the first model 
>and other studies that provide the adjusted R^2 for the other model?
>
>> Would it be possible to transform the adjusted R-square to make its 
>> distribution more normal, take its squareroot and then treat it as a 
>> correlation coefficient using rma()?
>
>No, it's not that simple. Just to give a counter-example: Say you have 
>a simple regression model of the form y = beta0 + beta1 x + e and take 
>the R^2 value from that model. Then sqrt(R^2) is equal to the *absolute 
>value* of the correlation between x and y. Since it is an absolute 
>value, it isn't going to behave like a regular/signed correlation coefficient and cannot be treated as such.
>
>Best,
>Wolfgang