<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>I have recently tried to finish all analyses and have realized
that I still haven't fully understood how moderation works here.
In the example below, I'm interested in whether the difference
between cor(X,Y) and cor(X,Z) is moderated by some factor. From my
reading of the metafor examples, var1var2X.Z:M should indicate
whether the cor(X,Z) is moderated by M. However, Wolfgang says
that term instead describes a moderation of the difference between
correlations. I'm perfectly happy to believe him, but I would
really like to understand why this is the case. <br>
</p>
<p>In my empirical example the results look as follows:</p>
<pre tabindex="0" class="GGHFMYIBMOB" id="rstudio_console_output" style="font-family: 'Lucida Console'; font-size: 10pt !important; outline: none; border: none; word-break: break-all; margin: 0px; -webkit-user-select: text; white-space: pre-wrap !important; line-height: 15px; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: -webkit-left; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);"> estimate se zval pval ci.lb ci.ub
var1var2X.Y 0.6580 0.0218 30.1818 <.0001 0.6152 0.7007 ***
var1var2X.Z -0.1282 0.0468 -2.7420 0.0061 -0.2199 -0.0366 **
var1var2Y.Z -0.2896 0.0480 -6.0315 <.0001 -0.3837 -0.1955 ***
M1 0.0736 0.0301 2.4474 0.0144 0.0147 0.1326 *
var1var2X.Z:M1 -0.1648 0.0786 -2.0969 0.0360 -0.3188 -0.0108 *
var1var2Y.Z:M1 -0.1266 0.0845 -1.4983 0.1340 -0.2922 0.0390
Here I'm interested in the difference between cor(x,z) and cor(y,z). Which of these interaction terms is critical to the moderation of this correlation, and why? Also, how can I interpret the direction of the effect?
If I look at another display of the same results, I get an understanding of the moderation, but I'm not sure about a clear test here. Would anova(res(0,1,-1,0,1,-1)) be a sensible test? What I'd really want to test would be if X.Z:M0 - Y.Z:M0 = X.Z:M1 - Y.Z:M1
estimate se zval pval ci.lb ci.ub
var1var2X.Y:M0 0.6580 0.0218 30.1818 <.0001 0.6152 0.7007 ***
var1var2X.Z:M0 -0.1282 0.0468 -2.7420 0.0061 -0.2199 -0.0366 **
var1var2Y.Z:M0 -0.2896 0.0480 -6.0315 <.0001 -0.3837 -0.1955 ***
var1var2X.Y:M1 0.7316 0.0207 35.3007 <.0001 0.6910 0.7722 ***
var1var2X.Z:M1 -0.2194 0.0483 -4.5437 <.0001 -0.3140 -0.1248 ***
var1var2Y.Z:M1 -0.3426 0.0491 -6.9767 <.0001 -0.4388 -0.2463 ***
</pre>
<div class="moz-cite-prefix">Am 22.08.2018 um 11:46 schrieb
Viechtbauer, Wolfgang (SP):<br>
</div>
<blockquote type="cite"
cite="mid:239114153aa740128f6b09223e2b87aa@UM-MAIL3213.unimaas.nl">
<pre class="moz-quote-pre" wrap="">Continuing with the earlier toy example:
Let's add a (study-level) moderator 'M' to the dataset:
dat2$dat$M <- c(2,2,2,4,4,4)
res <- rma.mv(yi, dat2$V, mods = ~ var1var2*M, data=dat2$dat)
res
I had to remove the 'random' part because with this little data, the full model isn't going to converge. The output includes:
Model Results:
estimate se zval pval ci.lb ci.ub
intrcpt 0.0600 0.3134 0.1914 0.8482 -0.5543 0.6743
var1var2X.Z 0.1100 0.3139 0.3504 0.7260 -0.5052 0.7252
var1var2Y.Z 0.5400 0.3419 1.5795 0.1142 -0.1301 1.2101
M 0.0700 0.1022 0.6847 0.4936 -0.1304 0.2704
var1var2X.Z:M -0.0050 0.1056 -0.0474 0.9622 -0.2119 0.2019
var1var2Y.Z:M -0.1100 0.1111 -0.9900 0.3222 -0.3278 0.1078
The line with 'var1var2X.Z:M' tells you to what extent the difference between cor(X,Y) and cor(X,Z) is moderated by M (and whether this moderation is significant).
Best,
Wolfgang
-----Original Message-----
From: Anna-Lena Schubert [<a class="moz-txt-link-freetext" href="mailto:anna-lena.schubert@psychologie.uni-heidelberg.de">mailto:anna-lena.schubert@psychologie.uni-heidelberg.de</a>]
Sent: Friday, 17 August, 2018 14:51
To: Michael Dewey; Viechtbauer, Wolfgang (SP); <a class="moz-txt-link-abbreviated" href="mailto:r-sig-meta-analysis@r-project.org">r-sig-meta-analysis@r-project.org</a>
Subject: Re: [R-meta] Comparing dependent, overlapping correlation coefficients
Dear Michael,
but how do I test if the effect of factor levels X vs Y (leaving out Z) is moderated by my continuous covariate? I believe that metafor gives me several ways to test for an overall interaction between a factor and a continuous covariate, but I'm not sure how I'd do follow-up comparisons from there.
Best,
Anna-Lena
Am 17.08.2018 um 14:44 schrieb Michael Dewey:
Dear Anna-Lena
The concept of interaction works generally for factor by factor, factor by covariate, and covariate by covariate. So you can just go ahead. I must say I have always found it harder to explain the covariate by covariate ones but that may be a defect in my explanatory powers.
Michael
On 17/08/2018 12:43, Anna-Lena Schubert wrote:
Dear Wolfgang,
thank you so much, this works perfectly well for me!
I have one final questions before I'm ready to analyze my data: Could I check whether this moderation by variable type is moderated by study characteristics? I.e., is there a way to include an interaction term that again specifically tests if moderator M moderates the difference in correlations between X and Y? I found your example on two categorial moderates and think I could apply that, but most of the moderators I'm thinking about are metric variables.
Best,
Anna-Lena
Am 15.08.2018 um 14:50 schrieb Viechtbauer, Wolfgang (SP):
library(metafor)
source(<a class="moz-txt-link-rfc2396E" href="https://gist.githubusercontent.com/wviechtb/700983ab0bde94bed7c645fce770f8e9/raw/5bb5601852b132af533aef41405d58a3ae04cf82/rmat.r">"https://gist.githubusercontent.com/wviechtb/700983ab0bde94bed7c645fce770f8e9/raw/5bb5601852b132af533aef41405d58a3ae04cf82/rmat.r"</a>)
dat <- read.table(header=TRUE, text = "
study var1 var2 ri ni
1 X Y .20 50
1 X Z .30 50
1 Y Z .52 50
2 X Y .34 35
2 X Z .43 35
2 Y Z .44 35")
dat2 <- rmat(ri ~ var1 + var2 | study, n=c(50,35), data=dat)
dat2
res <- rma.mv(yi, dat2$V, mods = ~ var1var2 - 1, random = ~ var1var2 | id, struct="UN", data=dat2$dat)
res
### three contrasts
anova(res, L=c(1, -1, 0))
anova(res, L=c(1, 0, -1))
anova(res, L=c(0, 1, -1))
You are interested in the first of these three contrasts, that is, whether cor(X,Y) = cor(X,Z).
Actually, I would recommend to work with:
dat2 <- rmat(ri ~ var1 + var2 | study, n=c(50,35), data=dat, rtoz=TRUE)
dat2
This applies Fisher's r-to-z transformation.
Best,
Wolfgang
-----Original Message-----
From: Anna-Lena Schubert [<a class="moz-txt-link-freetext" href="mailto:anna-lena.schubert@psychologie.uni-heidelberg.de">mailto:anna-lena.schubert@psychologie.uni-heidelberg.de</a>]
Sent: Wednesday, 15 August, 2018 11:08
To: Viechtbauer, Wolfgang (<a class="moz-txt-link-abbreviated" href="mailto:SP);r-sig-meta-analysis@r-project.org">SP);r-sig-meta-analysis@r-project.org</a>
Subject: Re: [R-meta] Comparing dependent, overlapping correlation coefficients
Hi Wolfgang,
thanks so much, I now believe I have sensible values in my V matrix.
They deviate slightly from manual calculations, but are really close.
I'm still lost on how to test for the interaction between X and Y then,
though. I tried
res <- rma.mv(yi, V, mods = ~ variable | studyID, data=dat,
method="ML"),
but the results don't correspond to the univariate meta-analyses I
conducted before. In addition, it tests (I believe) for a moderation of
all three correlations, while I'm only interested in the difference
between r_XY and r_XZ. Moreover, the manual says "In case the sampling
errors are correlated, then one can specify the entire
variance-covariance matrix of the sampling errors via the V argument",
but it seems I cannot simply leave "yi" out of the argument.
Best, Anna-Lena
Am 14.08.2018 um 22:18 schrieb Viechtbauer, Wolfgang (SP):
You do not need escalc(). The rmat() function gives you the variances along the diagonal of the 'V' matrix.
The variances should be (1 - ri^2)^2 / (ni - 1). You should be able to double-check that these values correspond to your data. Since ni should be the same for r_XY and r_XZ within a study, then it might be that the variances are roughly the same if the two correlations are not all that different. They should not be identical though (unless r_XY and r_XZ are the same).
Best,
Wolfgang
-----Original Message-----
From: R-sig-meta-analysis [<a class="moz-txt-link-freetext" href="mailto:r-sig-meta-analysis-bounces@r-project.org">mailto:r-sig-meta-analysis-bounces@r-project.org</a>] On Behalf Of Anna-Lena Schubert
Sent: Tuesday, 14 August, 2018 14:20
To: James Pustejovsky
<a class="moz-txt-link-abbreviated" href="mailto:Cc:r-sig-meta-analysis@r-project.org">Cc:r-sig-meta-analysis@r-project.org</a>
Subject: Re: [R-meta] Comparing dependent, overlapping correlation coefficients
Hi James,
I used Wolfgang's script on git to calculate the Cov(r_XY, r_XZ) by feeding it Cor(r_YZ). In the next step, I calculated Var(r_XY) and Var(r_XZ) by using the escalc function. However, Var(r_XY) always equals Var(r_XZ) for each study. Does this make sense?
I nevertheless added all three measures per study into a variance-covariance matrix such as:
r_XY r_XZ r_XY r_XZ
r_XY 0.004 0.0001 0 0
r_XZ 0.0001 0.004 0 0
r_XY 0 0 0.008 0.002
r_XZ 0 0 0.002 0.008
Then, I tried to feed everything into a multivariate meta-analysis:
res <- rma.mv(yi, V, mods = ~ variableType - 1, random = ~ variableType | studyNum, struct="UN", data=dat, method="ML")
The estimates I get for both of the correlation coefficients correspond closely to those I get when only meta-analyzing one of the variable types, which seems great. However, I'm still somewhat concerned that Var(r_XY) = Var(r_XZ). Do you think there may have been some mistake in my code or does it make sense that these variances are equal?
Best,
Anna-Lena
Am 10.08.2018 um 17:06 schrieb James Pustejovsky:
Anna-Lena,
The approach that you suggested (putting the data in "long" format and defining an indicator variable for whether Y or Z is the correlate) is just what I would recommend. However, there is a complication in that the estimates r_XY and r_XZ are correlated (correlated correlation coefficients...say that six times fast!), and the degree of correlation depends on r_YZ.
1) If you have extracted data on r_YZ then you could use this to compute Cov(r_XY, r_XZ) and then do a multivariate meta-analysis. See discussion here:
<a class="moz-txt-link-freetext" href="https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2018-January/000483.html">https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2018-January/000483.html</a>
And this function for computing the required covariance matrices:
<a class="moz-txt-link-freetext" href="https://gist.github.com/wviechtb/700983ab0bde94bed7c645fce770f8e9">https://gist.github.com/wviechtb/700983ab0bde94bed7c645fce770f8e9</a>
There are at least three further alternatives that might be simpler:
2) If you have r_YZ you could use it to compute the sampling variance of the difference between r_XY and r_XZ, that is:
Var(r_XY - r_XZ) = Var(r_XY) + Var(r_XZ) - 2 * Cov(r_XY, r_XZ)
You could then do a univariate meta-analysis on the difference between correlations.
3) If you do not have r_YZ then you won't be able to estimate Cov(r_XY, r_XZ) very well. You could make a guess about r_YZ and then follow approach (1) or (2) above, using cluster-robust variance estimation to account for the possibly mis-estimated sampling-variance covariance matrix.
4) Or you could ignore the covariance between r_XY and r_XZ entirely, fit the model to the long data as you describe above, and use cluster-robust variance estimation (clustering by sample) to account for the dependence between r_XY and r_XZ. This is the quickest and dirtiest approach, and the first thing I would try in practice before moving on to the more refined approaches above.
James
On Fri, Aug 10, 2018 at 9:21 AM Anna-Lena Schubert<a class="moz-txt-link-rfc2396E" href="mailto:anna-lena.schubert@psychologie.uni-heidelberg.de"><anna-lena.schubert@psychologie.uni-heidelberg.de></a> wrote:
Dear all,
I want to run a meta-analysis that compares dependent, overlapping
correlation coefficients (i.e., I want to see if X correlates more
strongly with Y than it does with Z). I already ran a meta-analysis
separately for both of these correlations and would now like to compare
those two pooled effect sizes statistically. Confidence intervals of the
two correlations do not overlap (r1 = .18 [.12; .24]; r2 = .32 [.25;
.39]), but I wonder if there may be a more elegant way to compare these
correlations than just based on CIs.
I wonder, for example, if a factorial variable could be used to identify
those correlations in a "long" data format style, and if I could test
for a significant interaction between variable type (Y vs. Z) and the
correlation in a meta-analysis:
Study Variable r
1 Y .20
1 Z .30
2 Y .34
2 Z .43
I would greatly appreciate if anyone could tell me if that's a good idea
or could recommend other approaches. Thanks in advance for any offers of
help!
Best,
Anna-Lena
</pre>
</blockquote>
<div class="moz-signature">-- <br>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Signatur</title>
<style type="text/css">
.auto-style1 {
font-weight: normal;
}
.auto-style2 {
font-weight: normal;
font-family: Arial;
border-bottom-style: solid;
border-bottom-width: 1px;
padding-bottom: 1px;
}
.auto-style3 {
font-family: Arial, Helvetica, sans-serif;
}
.auto-style4 {
font-size: small;
}
</style>
<h3 class="auto-style2">Dr. Anna-Lena Schubert</h3>
<p style="line-height: 1;"><img
src="cid:part1.62B0C85E.62FDFB3E@psychologie.uni-heidelberg.de"
data-filename="Logo_Diff.png" style="width: 215.5px; float:
right; height: 49.9136px;"><font class="auto-style3"><span
class="auto-style1"><span class="auto-style4">Postdoc at
Section of Personality</span><br class="auto-style4">
<span class="auto-style4">Heidelberg University - Institute
of Psychology</span></span></font></p>
<pre style="line-height: 1;">Hauptstraße 47-51
<span style="line-height: 1;">D-69117 Heidelberg
</span><span style="line-height: 1;">Germany</span></pre>
<pre style="line-height: 1;">Phone: +49 6221 54 7746
Mail: <a class="moz-txt-link-abbreviated" href="mailto:anna-lena.schubert@psychologie.uni-heidelberg.de">anna-lena.schubert@psychologie.uni-heidelberg.de</a>
Web: <a class="moz-txt-link-freetext" href="http://www.psychologie.uni-heidelberg.de/ae/diff/diff/people-schubert.html">http://www.psychologie.uni-heidelberg.de/ae/diff/diff/people-schubert.html</a></pre>
</div>
</body>
</html>