[R-meta] The P value of correlation coefficent in meta-analysis

Wed May 8 17:23:26 CEST 2024

Hi Guido and Lukasz,

Great thanks for the details! This issue was already extensively discussed in these publications, especially in the meta-analysis book. Thank you for helping working things out!

Cheers!
Pengzhen

---- Replied Message ----
| From | Lukasz Stasielowicz<lukasz.stasielowicz using uni-osnabrueck.de> |
| Date | 5/8/2024 03:25 |
| To | <r-sig-meta-analysis using r-project.org> |
| Cc | <maiqi1317 using 163.com> |
| Subject | Re: The P value of correlation coefficent in meta-analysis |
Hi Pengzhen,

Oh dear, your intuition is obviously correct. If we exclude
non-significant correlations, then we will overestimate the correlation.
In other words, we would get a biased effect estimate.

This issue is addressed in many meta-analytic textbooks, e.g.
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R.
(2021). Introduction to meta-analysis. John Wiley & Sons.

The authors offer a few free chapters, some of which could be useful in
your case.
https://introduction-to-meta-analysis.com/download/c01.pdf
For example, Figure 1.1 (p. 4) clearly shows that studies with
insignificant p-values are included in the meta-analysis.

The book also contains a chapter, "Vote counting - a new name for an old
problem," which has a nice example showing that even when all individual
studies have large p-values, the meta-analytic estimate can be
statistically significant. By combining the individual studies, we
increase the power and can detect even small effects.

Best,
Lukasz
--
Lukasz Stasielowicz
Osnabrück University
Institute for Psychology
Research methods, psychological assessment, and evaluation
Lise-Meitner-Straße 3
49076 Osnabrück (Germany)
Twitter: https://twitter.com/l_stasielowicz
Tel.: +49 541 969-7735

On 08.05.2024 03:30, r-sig-meta-analysis-request using r-project.org wrote:
Send R-sig-meta-analysis mailing list submissions to
r-sig-meta-analysis using r-project.org

To subscribe or unsubscribe via the World Wide Web, visit
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
or, via email, send a message with subject or body 'help' to
r-sig-meta-analysis-request using r-project.org

You can reach the person managing the list at
r-sig-meta-analysis-owner using r-project.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of R-sig-meta-analysis digest..."

Today's Topics:

1. Online course: Meta-analysis in R (info using physalia-courses.org)
2. The P value of correlation coefficent in meta-analysis
(Pengzhen Huang)
3. Correcting gain effects in nested studies (Zhouhan Jin)
4. Re: Correcting gain effects in nested studies (James Pustejovsky)

----------------------------------------------------------------------

Message: 1
Date: Tue, 7 May 2024 21:06:47 +0200 (CEST)
From: "info using physalia-courses.org" <info using physalia-courses.org>
To: r-sig-meta-analysis using r-project.org
Subject: [R-meta] Online course: Meta-analysis in R
Message-ID: <1715108807.553710121 using webmail.jimdo.com>
Content-Type: text/plain; charset="utf-8"

Dear all,
There are only 2 seats left for our upcoming online course, META-ANALYSIS IN R.
Dates: May 13-16, 2024
Online: Accessible internationally
This course covers:
Systematic review and meta-analysis process
Statistical analysis methods and interpretation
Model diagnostics and sensitivity analyses
Practical exercises with real meta-analytic datasets
Prerequisites include basic statistical knowledge and familiarity with R. Resources for R preparation are provided.
Course website: [ https://www.physalia-courses.org/courses-workshops/metain-r/ ]( https://www.physalia-courses.org/courses-workshops/metain-r/ )

Best regards,

Carlo

--------------------

Carlo Pecoraro, Ph.D

Physalia-courses DIRECTOR

info using physalia-courses.org

mobile: +49 17645230846

[[alternative HTML version deleted]]

------------------------------

Message: 2
Date: Wed, 8 May 2024 03:47:15 +0800 (GMT+08:00)
From: "Pengzhen Huang" <maiqi1317 using 163.com>
To: =?UTF-8?Q?r-sig-meta-analysis=40r-pr=E2=80=A6?=
<r-sig-meta-analysis using r-project.org>
Subject: [R-meta] The P value of correlation coefficent in
meta-analysis
Message-ID: <505c373a.1c5.18f54988e70.Coremail.maiqi1317 using 163.com>
Content-Type: text/plain; charset="utf-8"

Dear Community,

I submitted a meta-analysis paper months ago and am now dealing with the reviewers' comments. In my research, the Pearson correlation coefficients are considered as effect size and put into the meta-analysis, and we regard the coefficients representing to what extent two variables are correlated with each other.

On this point, one of reviewers argues that "as not all r values are significant, it does not make sense to put these non-significant correlation coefficients into the analysis".

I’m not sure how to reply to this reviewer’s comment. But I guess this may be a common issue in meta-analysis. May I have some advice from you or could you tell me some references I should read through?

I would greatly appreciate any suggestions you can provide!

All the best,
Pengzhen

[[alternative HTML version deleted]]

------------------------------

Message: 3
Date: Wed, 8 May 2024 00:14:11 +0000
From: Zhouhan Jin <zjin65 using uwo.ca>
To: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis using r-project.org>
Subject: [R-meta] Correcting gain effects in nested studies
Message-ID: <d68c25db-a085-4e10-9e4b-d23de85a24ee using Spark>
Content-Type: text/plain; charset="utf-8"

Hello All,

Hedges (2007) provides formulas for adjusting SMD effects (g) and their SEs for when primary studies have a nested design (below).

But I want to compute gain effects (ex. SMCC in metafor::escalc) from my nested studies, not SMDs.

So, how can I adjust my SMCCs and their SEs for nestedness in the primary studies?

adjusted_g =  g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
(n_cluster * n_bar - 2)))

adjusted_SE =  ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
( g^2 * (
(((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
(2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
)  )

Thanks a lot!

Best wishes,

Zhouhan

[[alternative HTML version deleted]]

------------------------------

Message: 4
Date: Tue, 7 May 2024 20:30:33 -0500
From: James Pustejovsky <jepusto using gmail.com>
To: Zhouhan Jin <zjin65 using uwo.ca>
Cc: R Special Interest Group for Meta-Analysis
<r-sig-meta-analysis using r-project.org>
Subject: Re: [R-meta] Correcting gain effects in nested studies
Message-ID:
<CAFUVuJyQmjrJJMFz3EZHGT+rR76YqKr1Zv6AVbvFqLcf+v_AOg using mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

See Taylor, Pigott, and Williams (2022;
https://doi.org/10.3102/0013189X211051319) for how to handle
cluster-randomized trials that involve gain scores or covariate adjustment.
They provide a shiny app too. The technical details are also described in
Appendix E of the What Works Clearinghouse handbook (Version 5;
https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-508.pdf).
See pp. 173-174

The methods described in these sources are consistent with the "general
recipe" for standardized mean difference estimates as described here:
https://www.jepusto.com/alternative-formulas-for-the-smd/

James

On Tue, May 7, 2024 at 7:14 PM Zhouhan Jin <zjin65 using uwo.ca> wrote:

Hello All,

Hedges (2007) provides formulas for adjusting SMD effects (g) and their
SEs for when primary studies have a nested design (below).

But I want to compute gain effects (ex. SMCC in metafor::escalc) from my
nested studies, not SMDs.

So, how can I adjust my SMCCs and their SEs for nestedness in the primary
studies?

adjusted_g =  g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
(n_cluster * n_bar - 2)))

adjusted_SE =  ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
( g^2 * (
(((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
(2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
)  )

Thanks a lot!

Best wishes,

Zhouhan

[[alternative HTML version deleted]]

------------------------------

Subject: Digest Footer

_______________________________________________
R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
To manage your subscription to this mailing list, go to:
https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

------------------------------

End of R-sig-meta-analysis Digest, Vol 84, Issue 10
***************************************************

	[[alternative HTML version deleted]]