[R-meta] Including subsections of test and overall test results in rma.mv

[Viechtbauer, Wolfgang (NP)]

Dear Yuhang,

Interesting question. I cannot give you a direct answer, but just some thoughts:

The 'overall' correlation is a so-called 'composite correlation' that can be reconstructed from the 4x4 correlation matrix for writing, reading, speaking, and whatever other variable these variables are being correlated with. For example, say that you have the following correlation matrix:

R <- structure(c(1, 0.4, 0.27, 0.27, 0.4, 1, 0.22, 0.54, 0.27, 
0.22, 1, 0.56, 0.27, 0.54, 0.56, 1), dim = c(4L, 4L))
rownames(R) <- colnames(R) <- c("writing", "reading", "speaking", "other")
R

Then the correlation between the sum (or mean) of the standardized writing, reading, and speaking variables with the "other variable" can be computed, for example, with the composite_r_matrix() function from the 'psychmeta' package:

library(psychmeta)
composite_r_matrix(R, 1:3, 4)

Or one can do this manually with:

sum(R[4,1:3]) / sqrt(sum(R[1:3,1:3]) * sum(R[4,4]))

So, there is a direct 'functional' relationship between the individual correlations and the overall one and in that sense, one might argue that including the individual correlations and the overall one is redundant. However, you are not including the writing-reading, writing-speaking, and reading-speaking correlations in your dataset and the relationship is non-linear. So in that sense, one might argue that including both sets is permissible. However, when doing so, it is important that one gets the covariance between all these correlations correct in the V matrix. You say that your V matrix captures those covariances, but I would be curious how you computed those covariances. Given the relationship above, it is of course possible to compute those covariances, but this doesn't seem entirely trivial to me.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Yuhang Hu via R-sig-meta-analysis
>Sent: Saturday, 08 April, 2023 5:39
>To: R meta
>Cc: Yuhang Hu
>Subject: [R-meta] Including subsections of test and overall test results in
>rma.mv
>
>Hello Meta Experts,
>
>I'm exploring the relation between a personality trait and an
>achievement test outcome across a set of studies.
>
>Some studies report the relation of the trait with both the overall
>achievement test outcome (one correlation) as well as the subsections of
>the test outcomes (multiple correlations).
>
>I'm interested in exploring the relationship mentioned above <<both>> in
>terms of the overall achievement test outcome as well as the subsections of
>the test outcomes.
>
>So, my current data looks like what I'm showing below.
>
>I do have a V matrix in my model that correlates the correlation coefs in
>each study due to the same subjects taking the subsections of the test
>outcomes <<as well as>> the overall test outcome.
>
>*My question is that: given my V matrix, is it fine if I include both the
>subsections of the test as well as the overall test outcomes in the same
>model?*
>
>(My hunch is that this is not permissible because in the current model the
>overall test outcome is essentially treated as a new outcome while the
>overall test outcome essentially subsumes the subsections of the test
>outcomes, not a new outcome.)
>
>rma.mv(r2z~test_outcome, V, random=~1 | trait_scale/study/test_outcome/es)
>
>study trait_scale test_outcome  r2z   v_r2z  es
>1         epq           overall
>1         epq           writing
>1         epq           reading
>1         epq           speaking
>2         16pf          overall
>3         epi            writing
>3         epi            speaking
>
>Thank you,
>Yuhang