[R-meta] Score Normalization for Moderator Analysis in Meta-Analysis

[Viechtbauer, Wolfgang (NP)]

Uhhh, no idea what happened here, but apparently the mailing list server thought that reply was so important to send it out three times ... Apologies for the spam.

Best,
Wolfgang

>-----Original Message-----
>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>Behalf Of Viechtbauer, Wolfgang (NP) via R-sig-meta-analysis
>Sent: Friday, 29 September, 2023 13:28
>To: Kiet Huynh
>Cc: Viechtbauer, Wolfgang (NP); R Special Interest Group for Meta-Analysis
>Subject: Re: [R-meta] Score Normalization for Moderator Analysis in Meta-Analysis
>
>I just noticed that the last question has remained unanswered:
>
>Depends on what you mean by "need". To run such an analysis, assuming 'scale' is
>just a two-level factor and you want to run a model with '~ factor(scale) *
>pompmean', then you will need five effect sizes, two for the first and three for
>the second level of 'scale'. That will give you just enough information to fit
>such a model and estimate the amount of residual heterogeneity.
>
>But I assume that this is not what you mean by "need". If you meant something
>along the lines of 'having enough power', then I cannot give you an answer to
>that question, because it is like asking: "I want to run a study - how many
>subjects do I need?" (although turns out that the answer to that question is:
>"three patients" -- https://www.youtube.com/watch?v=Hz1fyhVOjr4). To give an
>informed answer to that question, one would have to do a power analysis:
>
>Hedges, L. V., & Pigott, T. D. (2004). The power of statistical tests for
>moderators in meta-analysis. Psychological Methods, 9(4), 426-445.
>
>If you meant something along the lines of 'so that reviewers are not going to
>complain that my sample size is too small', then one could refer to rules of
>thumb like what you can find in the Cochrane Handbook:
>
>https://training.cochrane.org/handbook/current/chapter-10#section-10-11-5-1
>
>"It is very unlikely that an investigation of heterogeneity will produce useful
>findings unless there is a substantial number of studies. Typical advice for
>undertaking simple regression analyses: that at least ten observations (i.e. ten
>studies in a meta-analysis) should be available for each characteristic modelled.
>However, even this will be too few when the covariates are unevenly distributed
>across studies."
>
>To be clear, this is an entirely arbitrary rule (and one also finds suggestions
>like '5 studies per characteristic'). Also, what exactly 'for each characteristic
>modelled' means is not entirely clear, but say we interpret this as 'per model
>coefficient'. The model above has 4 model coefficients (including the intercept),
>so then we would need at least 40 effect sizes.
>
>To be fair, this rule does relate somewhat to the issue of overfitting, since
>more complex models require more data points to avoid overfitting. But even then,
>one would have to articulate more precisely what exactly one is concerned about.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: Kiet Huynh [mailto:kietduchuynh using gmail.com]
>>Sent: Thursday, 14 September, 2023 17:10
>>To: Viechtbauer, Wolfgang (NP)
>>Cc: R Special Interest Group for Meta-Analysis
>>Subject: Re: [R-meta] Score Normalization for Moderator Analysis in Meta-
>Analysis
>>
>>Hi Wolfgang,
>>
>>Thanks for the reminder about including links when cross posting.
>>
>>I appreciate the helpful expiation for the proportion/percentage of maximum
>>possible' (POMP) score method for moderation analysis. Especially helpful was
>the
>>tip on using the scale type to interact with the POMP score mean to determine if
>>the relationship between social support and the strength of the association
>>between LGBTQ+ discrimination and mental health differs depending on the scale
>>used. Do you have a sense of how many effect sizes would be needed for that?
>>
>>Best,
>>
>>Kiet