[R-meta] The P value of correlation coefficent in meta-analysis

Wed May 8 09:24:46 CEST 2024

Hi Pengzhen,

Oh dear, your intuition is obviously correct. If we exclude 
non-significant correlations, then we will overestimate the correlation. 
In other words, we would get a biased effect estimate.

This issue is addressed in many meta-analytic textbooks, e.g.
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. 
(2021). Introduction to meta-analysis. John Wiley & Sons.

The authors offer a few free chapters, some of which could be useful in 
your case.
https://introduction-to-meta-analysis.com/download/c01.pdf
For example, Figure 1.1 (p. 4) clearly shows that studies with 
insignificant p-values are included in the meta-analysis.

The book also contains a chapter, "Vote counting - a new name for an old 
problem," which has a nice example showing that even when all individual 
studies have large p-values, the meta-analytic estimate can be 
statistically significant. By combining the individual studies, we 
increase the power and can detect even small effects.

Best,
Lukasz
-- 
Lukasz Stasielowicz
Osnabrück University
Institute for Psychology
Research methods, psychological assessment, and evaluation
Lise-Meitner-Straße 3
49076 Osnabrück (Germany)
Twitter: https://twitter.com/l_stasielowicz
Tel.: +49 541 969-7735

On 08.05.2024 03:30, r-sig-meta-analysis-request using r-project.org wrote:
> Send R-sig-meta-analysis mailing list submissions to
> 	r-sig-meta-analysis using r-project.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
> 	https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> or, via email, send a message with subject or body 'help' to
> 	r-sig-meta-analysis-request using r-project.org
> 
> You can reach the person managing the list at
> 	r-sig-meta-analysis-owner using r-project.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of R-sig-meta-analysis digest..."
> 
> 
> Today's Topics:
> 
>     1. Online course: Meta-analysis in R (info using physalia-courses.org)
>     2. The P value of correlation coefficent in meta-analysis
>        (Pengzhen Huang)
>     3. Correcting gain effects in nested studies (Zhouhan Jin)
>     4. Re: Correcting gain effects in nested studies (James Pustejovsky)
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Tue, 7 May 2024 21:06:47 +0200 (CEST)
> From: "info using physalia-courses.org" <info using physalia-courses.org>
> To: r-sig-meta-analysis using r-project.org
> Subject: [R-meta] Online course: Meta-analysis in R
> Message-ID: <1715108807.553710121 using webmail.jimdo.com>
> Content-Type: text/plain; charset="utf-8"
> 
> 
> Dear all,
> There are only 2 seats left for our upcoming online course, META-ANALYSIS IN R.
> Dates: May 13-16, 2024
> Online: Accessible internationally
> This course covers:
> Systematic review and meta-analysis process
> Statistical analysis methods and interpretation
> Model diagnostics and sensitivity analyses
> Practical exercises with real meta-analytic datasets
> Prerequisites include basic statistical knowledge and familiarity with R. Resources for R preparation are provided.
> Course website: [ https://www.physalia-courses.org/courses-workshops/metain-r/ ]( https://www.physalia-courses.org/courses-workshops/metain-r/ )
>   
> Best regards,
>   
> Carlo
>   
>   
> 
> --------------------
> 
> Carlo Pecoraro, Ph.D
> 
> 
> Physalia-courses DIRECTOR
> 
> info using physalia-courses.org
> 
> mobile: +49 17645230846
> 
> 
> 
> 
> 
> 	[[alternative HTML version deleted]]
> 
> 
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Wed, 8 May 2024 03:47:15 +0800 (GMT+08:00)
> From: "Pengzhen Huang" <maiqi1317 using 163.com>
> To: =?UTF-8?Q?r-sig-meta-analysis=40r-pr=E2=80=A6?=
> 	 <r-sig-meta-analysis using r-project.org>
> Subject: [R-meta] The P value of correlation coefficent in
> 	meta-analysis
> Message-ID: <505c373a.1c5.18f54988e70.Coremail.maiqi1317 using 163.com>
> Content-Type: text/plain; charset="utf-8"
> 
> Dear Community,
> 
> 
> I submitted a meta-analysis paper months ago and am now dealing with the reviewers' comments. In my research, the Pearson correlation coefficients are considered as effect size and put into the meta-analysis, and we regard the coefficients representing to what extent two variables are correlated with each other.
> 
> 
> On this point, one of reviewers argues that "as not all r values are significant, it does not make sense to put these non-significant correlation coefficients into the analysis".
> 
> 
> I’m not sure how to reply to this reviewer’s comment. But I guess this may be a common issue in meta-analysis. May I have some advice from you or could you tell me some references I should read through?
> 
> 
> I would greatly appreciate any suggestions you can provide!
> 
> 
> 
> 
> All the best,
> Pengzhen
> 
> 
> 
> 
> 
> 
> 	[[alternative HTML version deleted]]
> 
> 
> 
> ------------------------------
> 
> Message: 3
> Date: Wed, 8 May 2024 00:14:11 +0000
> From: Zhouhan Jin <zjin65 using uwo.ca>
> To: R Special Interest Group for Meta-Analysis
> 	<r-sig-meta-analysis using r-project.org>
> Subject: [R-meta] Correcting gain effects in nested studies
> Message-ID: <d68c25db-a085-4e10-9e4b-d23de85a24ee using Spark>
> Content-Type: text/plain; charset="utf-8"
> 
> Hello All,
> 
> Hedges (2007) provides formulas for adjusting SMD effects (g) and their SEs for when primary studies have a nested design (below).
> 
> But I want to compute gain effects (ex. SMCC in metafor::escalc) from my nested studies, not SMDs.
> 
> So, how can I adjust my SMCCs and their SEs for nestedness in the primary studies?
> 
> 
> adjusted_g =  g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
>                    (n_cluster * n_bar - 2)))
> 
> adjusted_SE =  ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
>      ( g^2 * (
>        (((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
>           (2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
>          ((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
>      )  )
> 
> Thanks a lot!
> 
> Best wishes,
> 
> Zhouhan
> 
> 
> 	[[alternative HTML version deleted]]
> 
> 
> 
> ------------------------------
> 
> Message: 4
> Date: Tue, 7 May 2024 20:30:33 -0500
> From: James Pustejovsky <jepusto using gmail.com>
> To: Zhouhan Jin <zjin65 using uwo.ca>
> Cc: R Special Interest Group for Meta-Analysis
> 	<r-sig-meta-analysis using r-project.org>
> Subject: Re: [R-meta] Correcting gain effects in nested studies
> Message-ID:
> 	<CAFUVuJyQmjrJJMFz3EZHGT+rR76YqKr1Zv6AVbvFqLcf+v_AOg using mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> See Taylor, Pigott, and Williams (2022;
> https://doi.org/10.3102/0013189X211051319) for how to handle
> cluster-randomized trials that involve gain scores or covariate adjustment.
> They provide a shiny app too. The technical details are also described in
> Appendix E of the What Works Clearinghouse handbook (Version 5;
> https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-508.pdf).
> See pp. 173-174
> 
> The methods described in these sources are consistent with the "general
> recipe" for standardized mean difference estimates as described here:
> https://www.jepusto.com/alternative-formulas-for-the-smd/
> 
> James
> 
> On Tue, May 7, 2024 at 7:14 PM Zhouhan Jin <zjin65 using uwo.ca> wrote:
> 
>> Hello All,
>>
>> Hedges (2007) provides formulas for adjusting SMD effects (g) and their
>> SEs for when primary studies have a nested design (below).
>>
>> But I want to compute gain effects (ex. SMCC in metafor::escalc) from my
>> nested studies, not SMDs.
>>
>> So, how can I adjust my SMCCs and their SEs for nestedness in the primary
>> studies?
>>
>>
>> adjusted_g =  g * sqrt(1 - ((2 * (n_bar - 1) * icc) /
>>                    (n_cluster * n_bar - 2)))
>>
>> adjusted_SE =  ((Nt+Nc)/(Nt*Nc))*(1 + ((n_bar- 1)*icc)) +
>>      ( g^2 * (
>>        (((N_tot -2)*(1-icc)^2 ) + (n_bar*(N_tot - 2*n_bar)*icc^2) +
>>           (2* (N_tot - 2*n_bar) * icc * (1 - icc)) ) /
>>          ((2* (N_tot-2)) * ( (N_tot-2) - (2* (n_bar-1)*icc) ))
>>      )  )
>>
>> Thanks a lot!
>>
>> Best wishes,
>>
>> Zhouhan
>>
>>
> 
> 	[[alternative HTML version deleted]]
> 
> 
> 
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis
> 
> 
> ------------------------------
> 
> End of R-sig-meta-analysis Digest, Vol 84, Issue 10
> ***************************************************