[R-meta] [External] RE: Effect Sizes and Beta Coefficients

Mon Jun 17 17:12:46 CEST 2024

Dear Rebecca

In general estimates adjusted for different sets of covariates are not 
directly comparable. If the sets are almost the same then you may be 
able to argue for including them. It might also depend on strategic 
questions like how many studies you already have and whether adding one 
more is worth the hassle of explaining why you included it.

In short as so often the answer is, how brave do you feel?

Michael

On 17/06/2024 15:27, Hall, Rebecca via R-sig-meta-analysis wrote:
> Dear Wolfgang,
> 
> Many thanks for your assistance so far. We are still looking at papers 
> using various statistical analyses, and have come across one which 
> utilises a one-way MANCOVA. Please could you advise on the validity of 
> calculating an effect size from this type of analysis, as the reported F 
> and p values are altered by controlling for covariates, whereas this has 
> not been the case in other studies.
> 
> Similarly, we have a paper reporting odds ratios which, if we are 
> correct, we can directly convert to an effect size (pearson's r). 
> However, the odds ratios in question are presented as two models, one 
> with more covariates than the other. Is there a particular model you 
> would suggest including (see below)?
> 
> 
> Best,
> Rebecca
> 
> *Dr Rebecca Hall | Research Associate
> *
> Department of Psychology | Lancaster University
> 
> ------------------------------------------------------------------------
> *From:* Viechtbauer, Wolfgang (NP) 
> <wolfgang.viechtbauer using maastrichtuniversity.nl>
> *Sent:* 13 June 2024 13:43
> *To:* Hall, Rebecca <r.hall5 using lancaster.ac.uk>; R Special Interest Group 
> for Meta-Analysis <r-sig-meta-analysis using r-project.org>
> *Subject:* RE: [External] RE: [R-meta] Effect Sizes and Beta Coefficients
> Dear Rebecca,
> 
> Please see below for my answer.
> 
> Best,
> Wolfgang
> 
>> -----Original Message-----
>> From: Hall, Rebecca <r.hall5 using lancaster.ac.uk>
>> Sent: Wednesday, June 12, 2024 17:56
>> To: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>; R
>> Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-project.org>
>> Subject: Re: [External] RE: [R-meta] Effect Sizes and Beta Coefficients
>>
>> Dear Wolfgang,
>>
>> Many thanks for your response. Am I correct in thinking that I can convert a p-
>> value in R using qt(1- p-value, n-1)?
> 
> I assume you are asking about a p-value from a regression model. First 
> of all, this is typically a two-sided p-value, so the given p-value must 
> first be divided by 2. Also, the degrees of freedom are n-p-1 for a 
> regression model with p regression coefficients. So it would be:
> 
> qt(1-pval/2, df=n-p-1)
> 
> Note that the (two-sided) p-value does not tell you anything about the 
> sign of the regression coefficient (which in turn corresponds to the 
> sign of the test statistic and hence the sign of the (semi)partial 
> correlation), so additional information reported needs to be used to 
> determine the correct sign.
> 
>> Further to this, please could you advise how I might then calculate the partial
>> correlation from a one-sample t-test?
> 
> You can't. A one-sample t-test is used to test if the mean of a single 
> variable differs significantly from some value. It is not possible to 
> compute the partial correlation for a single variable.
> 
>> Best,
>> Rebecca
>>
>> Dr Rebecca Hall | Research Associate
>> Department of Psychology | Lancaster University
>>
>> ________________________________________
>> From: Viechtbauer, Wolfgang (NP) <wolfgang.viechtbauer using maastrichtuniversity.nl>
>> Sent: 11 June 2024 17:21
>> To: R Special Interest Group for Meta-Analysis <r-sig-meta-analysis using r-
>> project.org>
>> Cc: Hall, Rebecca <r.hall5 using lancaster.ac.uk>
>> Subject: [External] RE: [R-meta] Effect Sizes and Beta Coefficients
>>
>> This email originated outside the University. Check before clicking links or
>> attachments.
>>
>> Dear Rebecca,
>>
>> This is my personal opinion: I would consider this approach outdated.
>>
>> Typically, in a situation like this, one also knows the t-statistic for the
>> coefficient of interest (or its p-value from which one can back-calculate the t-
>> statistic). In that case, one can compute the (semi)partial correlation
>> coefficient for the coefficient.
>>
>> However, whether one should combine such 'partial' effect sizes with bivariate
>> correlations is debatable in the first place. A relevant article that
>> essentially argues against this is:
>>
>> Aloe, A. M. (2015). Inaccuracy of regression results in replacing bivariate
>> correlations. Research Synthesis Methods, 6(1), 21-27.
>> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoi.org%2F10.1002%2Fjrsm.1126&data=05%7C02%7Challr4%40live.lancs.ac.uk%7C2ce13474f1ec470a8a9c08dc8ba66749%7C9c9bcd11977a4e9ca9a0bc734090164a%7C0%7C0%7C638538794019673955%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=OUXIjY07mR54gO5eHMgodzvyAuc3uB4CnWfESXhYNJA%3D&reserved=0 <https://doi.org/10.1002/jrsm.1126>
>>
>> At least, one could try to capture some of the heterogeneity introduced by this
>> by including a moderator in the model that indicates the type of correlation
>> coefficient. With enough studies, one could even go a step further and include
>> moderators that indicate which covariates were included in the original
>> regression models from which the (semi)partial correlations were obtained (as a
>> bunch of dummy variables).
>>
>> Best,
>> Wolfgang
>>
>> > -----Original Message-----
>> > From: R-sig-meta-analysis <r-sig-meta-analysis-bounces using r-project.org> On
>> Behalf
>> > Of Hall, Rebecca via R-sig-meta-analysis
>> > Sent: Tuesday, June 11, 2024 17:57
>> > To: r-sig-meta-analysis using r-project.org
>> > Cc: Hall, Rebecca <r.hall5 using lancaster.ac.uk>
>> > Subject: [R-meta] Effect Sizes and Beta Coefficients
>> >
>> > Dear all,
>> >
>> > I have a question regarding the use of a beta coefficient as a substitute for
>> > effect size where a paper lacks statistical data for Pearson's r to otherwise
>> be
>> > calculated.
>> >
>> > Peterson & Brown (2005) support the use of standardised beta coefficients and
>> > relative SE in the place of correlations, but Roth et al. (2018) criticise
>> this.
>> > I'm therefore wondering whether there is a general consensus regarding the use
>> > of beta coefficients, and should Peterson & Brown's approach no longer be
>> > appropriate then I would be gladly advised on the alternative method that
>> should
>> > be utilised.
>> >
>> > Many thanks,
>> > Rebecca
>> >
>> > Dr Rebecca Hall | Research Associate
>> > Department of Psychology | Lancaster University
> 
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>	Virus-free.www.avg.com <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> 
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
> 
> _______________________________________________
> R-sig-meta-analysis mailing list @ R-sig-meta-analysis using r-project.org
> To manage your subscription to this mailing list, go to:
> https://stat.ethz.ch/mailman/listinfo/r-sig-meta-analysis

-- 
Michael