[R-meta] effect size for interaction from mixed designs

[Viechtbauer, Wolfgang (NP)]

Please see below for my responses.

Best,
Wolfgang

>-----Original Message-----
>From: Filippo Gambarota [mailto:filippo.gambarota using gmail.com]
>Sent: Monday, 08 August, 2022 17:41
>To: Viechtbauer, Wolfgang (NP)
>Cc: R meta
>Subject: Re: [R-meta] effect size for interaction from mixed designs
>
>Thank you as always Wolfgang.
>I'm wondering if this could work. Assuming that all studies used a common measure
>(let's say reaction times on a computer task). Ideally, If I have the interaction
>parameter and the standard error (e.g., from a mixed model output) I could
>directly use that measure.

Correct.

>Alternatively and following your suggestion I should standardize the numerator
>using the same quantity ( sqrt(MSE) ) but this quantity is influenced by the
>repeated measure design (i.e., the correlation).

Correct, the MSE will have a different 'meaning' depending on the design. Take a one-way ANOVA with a two-level between versus within subject factor. For the same data, the MSE will be different depending on the design. So while the mean difference between the two groups/conditions is the same, once we standardize based on the MSE we might end up with rather different values.

>Thank you
>
>On Mon, 8 Aug 2022 at 14:53, Viechtbauer, Wolfgang (NP)
><wolfgang.viechtbauer using maastrichtuniversity.nl> wrote:
>Dear Filippo,
>
>yes, measures like eta-squared or omega-squared are typically not a good choice
>for a meta-analysis as they are directionless. As you found, one can compute
>effect sizes also for interactions akin to Cohen's d. But yes, the type of design
>does matter (in particular, with respect to how the sampling variance needs to be
>computed). What I describe under that link assumes we are dealing with 2x2 ANOVAs
>with both factors being 'between-subject' factors. Once you have a within-subject
>factor (either one or two of them), then the correlation between the conditions
>will start to play a role again. Another issue here is what we put into the
>denominator when standardizing. In the between-subject case above, the
>computations assume that we are using sqrt(MSE) from the model. If one wants to
>combine such effects with those from mixed/fully within designs, one needs to do
>something analogous there, as otherwise the effects are not comparable.
>
>I think James Pustejovksy has something on his blog (https://www.jepusto.com)
>where he describes a very general method for computing d-type effect sizes. I
>assume this would also cover the cases above.
>
>Best,
>Wolfgang
>
>>-----Original Message-----
>>From: R-sig-meta-analysis [mailto:r-sig-meta-analysis-bounces using r-project.org] On
>>Behalf Of Filippo Gambarota
>>Sent: Monday, 08 August, 2022 14:17
>>To: R meta
>>Subject: [R-meta] effect size for interaction from mixed designs
>>
>>Hello!
>>I'm planning to conduct a meta-analysis focusing on interaction
>>effects. Firstly I was thinking about using an interaction-specific
>>effect size such as eta-squared or omega-squared. After reading some
>>papers and some other posts
>>(https://stat.ethz.ch/pipermail/r-sig-meta-analysis/2018-February/000658.html)
>>now it's clear that I can simply use a cohen's d like measure to
>>standardize the difference between the two simple effects. However,
>>I'm wondering if the type of ANOVA-like design matters in terms of
>>fully between/within or mixed. In fact, I'm pretty sure that I'll
>>encounter different designs from published studies.
>>Thank you!
>>--
>>Filippo Gambarota
>>PhD Student - University of Padova
>>Department of Developmental and Social Psychology
>>Website: filippogambarota
>>Research Group: Colab   Psicostat