[R-sig-ME] Help with determining effect sizes

[Maarten Jung]

Dear Francesco,

I don't think there is a "standard" way to calculate effect sizes for
linear mixed models due to the way the variance is partitioned (see
e.g. [1]).
One way to compute something similar to Cohen's d would be to divide
the difference between the estimated means of two conditions by a
rough estimate of the standard deviation of the response variable
which you can get by
sd(predict(your_model_name))

Best,
Maarten

[1] https://afex.singmann.science/forums/topic/compute-effect-sizes-for-mixed-objects#post-295


On Sat, Oct 5, 2019 at 10:01 AM Francesco Romano <fbromano77 using gmail.com> wrote:
>
> Dear all,
>
> A journal has asked that I determine the effect sizes for a series of
> dummy-coded contrasts from the following ME model:
>
> RT ~ Group * Grammaticality + (1 + Grammaticality | Participant) +
>     (1 + Group | item)
>
> Here RT is my continuous outcome variable measured in milliseconds, Group
> is a factor with 3 levels (NS, L2, and HL), and Grammaticality a factor
> with 2 levels (gr and ungr). After relevelling —NOTE: I am deliberately
> omitting the call for each new relevelled model here— I obtained a series
> of contrasts which are tabulated below (not sure you can view this whole):
>
>
> Reference level
>
> Contrasts
>
> Estimate
>
> (ms)
>
> Effect size
>
> (Cohen’s *d*)
>
> SE
>
> df
>
> *t*
>
> *p*
>
> HL
>
> GR vs UNGR
>
>  -213
>
>
>
> 89
>
> 72.13
>
> -2.399
>
> < .05*
>
> L2
>
> GR vs UNGR
>
>  -408
>
>
>
> 90
>
> 74.18
>
> -4.513
>
> < .001***
>
> L1
>
> GR vs UNGR
>
> -111
>
>
>
> 73
>
> 70.02
>
> -1.520
>
> > .05
>
>
>
> HL > L2
>
>  -25
>
>
>
> 191
>
> 43.48
>
> -.135
>
> > .05
>
> GR
>
> L1 > HL
>
>  400
>
>
>
> 175
>
> 43.81
>
> 2.286
>
> < .05*
>
>
>
> L1 > L2
>
>  374
>
>
>
> 179
>
> 43.59
>
> 2.092
>
> < .05*
>
>
>
> HL > L2
>
>  -219
>
>
>
> 179
>
> 42.70
>
> -1.226
>
> > .05
>
> UNGR
>
> L1 > HL
>
>  298
>
>
>
> 164
>
> 43
>
> 1.817
>
> > .05
>
>
>
> L1> L2
>
> 77
>
>
>
> 166
>
> 42.03
>
> .469
>
> > .05
>
> How would I go about determining the Cohen's *d* for each of the contrasts?
>
> The model call is:
>
> Linear mixed model fit by REML. t-tests use Satterthwaite's method
> ['lmerModLmerTest']
> Formula: RT ~ Group * Grammaticality + (1 + Grammaticality | Participant) +
>
>     (1 + Group | item)
>    Data: RTanalysis
>
> REML criterion at convergence: 52800
>
> Scaled residuals:
>     Min      1Q  Median      3Q     Max
> -2.1696 -0.6536 -0.1654  0.5060  5.0134
>
> Random effects:
>  Groups      Name               Variance Std.Dev. Corr
>  item        (Intercept)         71442   267.29
>              GroupL2              1144    33.82    0.80
>              GroupNS              9951    99.76   -0.43 -0.88
>  Participant (Intercept)        235216   484.99
>              Grammaticalityungr  50740   225.25   -0.39
>  Residual                       378074   614.88
> Number of obs: 3342, groups:  item, 144; Participant, 46
>
> Fixed effects:
>                            Estimate Std. Error      df t value Pr(>|t|)
> (Intercept)                 2801.98     136.70   48.85  20.498   <2e-16 ***
> GroupL2                      -25.86     191.20   43.48  -0.135   0.8931
> GroupNS                     -400.63     175.22   43.81  -2.286   0.0271 *
> Grammaticalityungr          -213.87      89.17   72.13  -2.399   0.0190 *
> GroupL2:Grammaticalityungr  -194.57     107.25   42.55  -1.814   0.0767 .
> GroupNS:Grammaticalityungr   102.31      99.39   43.45   1.029   0.3090
> ---
> Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>
> Correlation of Fixed Effects:
>             (Intr) GropL2 GropNS Grmmtc GrL2:G
> GroupL2     -0.672
> GroupNS     -0.744  0.526
> Grmmtcltyng -0.404  0.222  0.260
> GrpL2:Grmmt  0.259 -0.391 -0.205 -0.589
> GrpNS:Grmmt  0.299 -0.202 -0.392 -0.702  0.540
> convergence code: 0
> Model failed to converge with max|grad| = 0.0477764 (tol = 0.002, component
> 1)
>
> The distribution of the outcome is fairly normal and the overall mean,
> without considering the two fixed effects, is very close to the means of
> each of the three groups (without considering the effect of Grammaticality)
> as well as the means of the two levels of grammaticality (without
> considering the effect of group).
>
> The package simR can simulate data to determine power, amongst other things,
> but I am not sure how to do this for models with interactions such as mine.
>
> Use of simR is recommended in Brysbaert and Stevens (2018)
> https://www.journalofcognition.org/articles/10.5334/joc.10/. Perhaps there
> is a simpler way of extracting *d *from the stats I already know?
>
> Any help would be greatly appreciated,
>
> Francesco
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models