[R-meta] Meta-Analysis using different correlation coefficients

[Lukasz Stasielowicz]

Dear Lena,

the order of the sentences in my original reply could be confusing, so 
just to clarify:
I meant that it is generally possible to transform effect sizes and many 
meta-analysts choose this option. However, I wasn't suggesting that this 
approach is better/equivalent/inferior. If you choose this option, then 
point-biserial correlation is mathematically equivalent to Pearson 
correlation. Hence, there would be no need to transform the correlation.

However, I've mentioned that there are arguments against transforming 
effect sizes. I should have added that it also means that one could 
argue that point-biserial correlation and Pearson correlation shouldn't 
be considered together in one meta-analysis. In general, "when in doubt 
follow Wolfgang's advice" is a good heuristic. In particular, if you 
don't have a strong opinion on a certain topic.

There are many decisions that can be criticized during peer-review in 
primary studies (why ANOVA and not multi-level models? why multi-level 
models and not structural equation modelling?) and meta-analysis (e.g., 
dealing with dependent effect sizes, inclusion of regression 
coefficients when sythesizing correlations, inclusion of studies without 
weighting/post-stratification). While there are some decisions on which 
most of us would agree there are also areas where the opinions may vary 
(or: some reviewers may have strong opinions and other can be 
indifferent). Transformation/inclusion of other effect sizes belongs to 
the second category.

In your case, the safest solution is to restrict meta-analysis to only 
one type of effect sizes (e.g. Pearson correlation). You coud cite the 
article (Jacobs & Viechtbauer) to justify exclusion of other effect sizes.
However, if this means that say only 5 studies meet your inclusion 
criteria, then one could argue that including/transforming all effect 
sizes is justifiable (more information, moderator analyses etc). One 
could conduct a moderator analysis (effect size type, see previous 
message) in order to test the robustness of the results and address 
potential concerns of reviewers/readers.
There are many meta-analysts, who would consider all effect sizes 
irrespective of the number of available studies but it is risky as some 
reviewers could criticize the lack of justification. Providing a 
justification could help, but in (rare?) cases it will be rejected.


Best,
Lukasz
-- 
Lukasz Stasielowicz
Osnabrück University
Institute for Psychology
Research methods, psychological assessment, and evaluation
Seminarstraße 20
49074 Osnabrück (Germany)

Am 21.02.2022 um 14:04 schrieb Lena Pollerhoff:
> Dear Lukasz,
> 
> Thank you so much for answering, we really appreciate the effort and 
> your time.
> 
> We had also done some research, which lead us to the following thread 
> from Wolfgang Viechtbauer: 
> https://chat.stackexchange.com/rooms/60238/discussion-between-mark-white-and-wolfgang 
> <https://chat.stackexchange.com/rooms/60238/discussion-between-mark-white-and-wolfgang>, 
> and further to his paper (Jacobs & Viechtbauer) regarding point-biserial 
> and biserial correlations („Estimation of the biserial correlation and 
> its sampling variance for use in meta-analysis“) from 2016 
> (https://onlinelibrary.wiley.com/doi/full/10.1002/jrsm.1218 
> <https://onlinelibrary.wiley.com/doi/full/10.1002/jrsm.1218>).
> 
> Based on this we were a little worried about including point-biserial 
> and Pearson’s product-moment correlations in the same meta-analysis 
> without transforming the point-biserial coefficients? E.g., in the paper 
> they are saying „Unlike the point-biserial correlation coefficient, 
> biserial coefficients can therefore be integrated with product-moment 
> correlation coefficients in the same meta-analysis“, which lead us to 
> assume (contrary to your suggestion) that point-biserial and 
> product-moment correlation coefficients should not be included in the 
> same meta-analysis without further transformation? But the paper also 
> exclusively treats cases where a variable was *artificially* 
> dichotomized (and still analyzed with point-biserial correlation instead 
> of biserial correlation).
> 
> But you are suggesting that in case the point-biserial correlation is 
> based on a naturally occurring binary variable (e.g., gender), we can 
> integrate it with Pearson's poruduct-moment correlation in the same 
> analysis?
> 
> Thank you so much in advance and best wishes
> Lena
> 
>> Am 15.02.2022 um 08:03 schrieb Lukasz Stasielowicz 
>> <lukasz.stasielowicz using uni-osnabrueck.de 
>> <mailto:lukasz.stasielowicz using uni-osnabrueck.de>>:
>>
>> Dear Lena,
>>
>> since you haven't received a response yet, I will address some points:
>>
>> It is important to define a target effect size for the meta-analysis, 
>> e.g. Pearson correlation r. If other effect sizes are reported in the 
>> primary studies then one could transform them or use the raw data to 
>> compute the target effect size.
>>
>> If Pearson correlation is the target effect size and you have 
>> point-biserial correlations then no transformation is necessary 
>> because it is mathematically equivalent to Pearson correlation.
>>
>> Spearman correlation and Pearson correlation will not always lead to 
>> similar values, as the latter is less likely to detect nonlinear 
>> (monotonic) relationships. Therefore, transforming Spearman 
>> correlations to Pearson correlations could ensure, that the effect 
>> size values can be interpreted the same way. In this particular case 
>> one could use arcsin formulas:
>> de Winter, J. C. F., Gosling, S. D., & Potter, J. (2016). Comparing 
>> the Pearson and Spearman correlation coefficients across distributions 
>> and sample sizes: A tutorial using simulations and empirical data. 
>> Psychological Methods, 21(3), 273–290. doi.org/10.1037/met0000079 
>> <http://doi.org/10.1037/met0000079>
>> You'll find formulas for other transformations in textbooks and other 
>> journal articles.
>>
>> While many meta-analysts tend to use all available information and 
>> apply all possible transformations (d --> r, Spearman --> Pearson, OR 
>> --> r), there are some people, who would point out that under certain 
>> circumstances not all transformations are valid (e.g., d --> r  can 
>> the distinction between control group and experimental group be 
>> thought as a part of an underlying continuous distribution or is it a 
>> natural dichotomy?).
>> However, some people would argue that every effect size is useful (in 
>> particular, if the meta-analysis is small).
>> Sometimes meta-analysts conduct a moderator analysis and compare 
>> converted effect sizes (e.g., d-->r) with effect sizes that didn't 
>> require converting (e.g. r), in order to check the influence of 
>> pooling different designs/effects.
>>
>> Multiple effect sizes: If there are multiple studies with multiple 
>> effect sizes then a three-level meta-analysis could be useful. 
>> Alternatively, one could use cluster-robust variance estimation, e.g. 
>> cran.r-project.org/web/packages/clubSandwich/vignettes/meta-analysis-with-CRVE.html 
>> <http://cran.r-project.org/web/packages/clubSandwich/vignettes/meta-analysis-with-CRVE.html>
>> If there is only one sample with multiple effects then choosing one 
>> effect size (e.g., the typical operationalization of the constructs, 
>> the most valid operationalization) could be an option. Alternatively, 
>> one could compute a mean effect size, as you have mentioned in your 
>> message.
>>
>>
>> I hope it answers at least some of your questions. Good luck with your 
>> project!
>>
>>
>> Best,
>> Lukasz
>> -- 
>> Lukasz Stasielowicz
>> Osnabrück University
>> Institute for Psychology
>> Research methods, psychological assessment, and evaluation
>> Seminarstraße 20
>> 49074 Osnabrück (Germany)
>>
>> Am 09.02.2022 um 07:54 schrieb 
>> r-sig-meta-analysis-request using r-project.org 
>> <mailto:r-sig-meta-analysis-request using r-project.org>:
>>> Send R-sig-meta-analysis mailing list submissions to
>>> r-sig-meta-analysis using r-project.org 
>>> <mailto:r-sig-meta-analysis using r-project.org>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>>> or, via email, send a message with subject or body 'help' to
>>> r-sig-meta-analysis-request using r-project.org
>>> You can reach the person managing the list at
>>> r-sig-meta-analysis-owner using r-project.org
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of R-sig-meta-analysis digest..."
>>> Today's Topics:
>>>    1. Meta-Analysis using different correlation coefficients
>>>       (Lena Pollerhoff)
>>>    2. Meta-Analysis using different correlation coefficients
>>>       (Lena Pollerhoff)
>>>    3. Comparison to baseline- remove intercept or keep? (Danielle Hiam)
>>>    4. questions on some functions in metafor and clubsandwich
>>>       (Brendan Hutchinson)
>>> ----------------------------------------------------------------------
>>> Message: 1
>>> Date: Tue, 8 Feb 2022 13:39:52 +0100
>>> From: Lena Pollerhoff <lena using pollerhoff.de>
>>> To: r-sig-meta-analysis using r-project.org
>>> Subject: [R-meta] Meta-Analysis using different correlation
>>> coefficients
>>> Message-ID: <AA0B9D07-6B89-4BF7-91F7-D32B5FDE031A using pollerhoff.de>
>>> Content-Type: text/plain; charset="utf-8"
>>> Hello,
>>> We are currently conducting a meta-analysis based on correlation 
>>> coefficients.
>>> We have received a huge amount of raw datasets, so that we are able 
>>> to calculate effect sizes/correlations coefficients on our own for 
>>> many datasets, and  we have other correlations extracted from the 
>>> original pubs. Therefore I have a couple of questions:
>>> 1. If one variable is dichotomous and the other variable is 
>>> continuous but not normally distributed, what kind of coefficient 
>>> should be calculated? We’d go for point-biserial if the variable is 
>>> naturally dichotomous (not artificially dichotomized), and for 
>>> biserial correlation if the dichotomous variable was artificially 
>>> dichotomized, but are worried that both require normal distribution 
>>> of the continuous variable?
>>> 2. We are wondering how to best integrate person’s product moment 
>>> correlation coefficients (both continuous, normally distributed 
>>> variables), (point-) biserial correlation coefficients (for 1 
>>> (artificial) dichotomous and 1 continuous variable) and spearman rang 
>>> correlation coefficients (for non-parametric, both continuous 
>>> variables) in one meta-analysis? Just use the raw values? Or is it 
>>> better to transform them in a homogenous way (I’ve read Fisher’s z 
>>> makes less sense for anything else than Pearson’s r as a 
>>> variance-stabilizing procedure?)? Can spearman rho be converted using 
>>> fisher’s z transformation? I’ve also read that it is not advisable to 
>>> include product-moment correlation and point-biserial correlation in 
>>> one meta-analysis, is there a way to convert the point-biserial 
>>> correlation to something that can be integrated with Pearson’s r and 
>>> Spearman’s rho?
>>> 3. I have multiple effect sizes within one sample and I want to 
>>> aggregate them, how do I define rho in the aggregate function from 
>>> the metafor package? Is it possible to calculate rho based on the raw 
>>> datasets? Or would it better to think in a conservative way and 
>>> assume perfect redundancy (i.e., rho = 0.9)?
>>> Thanks in advance for your time and effort!
>>> Best,
>>> Lena
>>>
>>
>> _______________________________________________
>> R-sig-meta-analysis mailing list
>> R-sig-meta-analysis using r-project.org 
>> <mailto:R-sig-meta-analysis using r-project.org>
>> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
>