[R-sig-ME] Help with determining effect sizes

Maarten Jung M@@rten@Jung @end|ng |rom m@||box@tu-dre@den@de
Sat Oct 5 21:57:37 CEST 2019 

Hi Francesco,

You already got the unstandardized effect sizes - these simply are the
estimated reaction time differences ("Contrasts Estimate (ms)" in your
table).
If you standardize these differences by dividing by a measure of variation,
e.g. an estimate of the standard deviation of your response variable, you
get so called standardized effect sizes. I guess the latter is what your
reviewer is looking for but (as also mentioned in [1]) one can argue that
this standardization has its own pitfalls.

Best,
Maarten

On Sat, 5 Oct 2019, 21:05 Francesco Romano <fbromano77 using gmail.com> wrote:

> Hi Maarten,
>
> Thank you so much for your suggestions. In the source that you
> recommended, a paper is mentioned that advises reporting unstandardised
> effect sizes but I’m not sure how to do that. Is it the same as the
> division you explained in your message? If so, how would I obtain the
> estimated means for the specific contrasts in my table?
>
> Francesco
>
> On Sat, Oct 5, 2019 at 12:13 PM Maarten Jung <
> Maarten.Jung using mailbox.tu-dresden.de> wrote:
>
>> Dear Francesco,
>>
>> I don't think there is a "standard" way to calculate effect sizes for
>> linear mixed models due to the way the variance is partitioned (see
>> e.g. [1]).
>> One way to compute something similar to Cohen's d would be to divide
>> the difference between the estimated means of two conditions by a
>> rough estimate of the standard deviation of the response variable
>> which you can get by
>> sd(predict(your_model_name))
>>
>> Best,
>> Maarten
>>
>> [1]
>> https://afex.singmann.science/forums/topic/compute-effect-sizes-for-mixed-objects#post-295
>>
>>
>> On Sat, Oct 5, 2019 at 10:01 AM Francesco Romano <fbromano77 using gmail.com>
>> wrote:
>> >
>> > Dear all,
>> >
>> > A journal has asked that I determine the effect sizes for a series of
>> > dummy-coded contrasts from the following ME model:
>> >
>> > RT ~ Group * Grammaticality + (1 + Grammaticality | Participant) +
>> >     (1 + Group | item)
>> >
>> > Here RT is my continuous outcome variable measured in milliseconds,
>> Group
>> > is a factor with 3 levels (NS, L2, and HL), and Grammaticality a factor
>> > with 2 levels (gr and ungr). After relevelling —NOTE: I am deliberately
>> > omitting the call for each new relevelled model here— I obtained a
>> series
>> > of contrasts which are tabulated below (not sure you can view this
>> whole):
>> >
>> >
>> > Reference level
>> >
>> > Contrasts
>> >
>> > Estimate
>> >
>> > (ms)
>> >
>> > Effect size
>> >
>> > (Cohen’s *d*)
>> >
>> > SE
>> >
>> > df
>> >
>> > *t*
>> >
>> > *p*
>> >
>> > HL
>> >
>> > GR vs UNGR
>> >
>> >  -213
>> >
>> >
>> >
>> > 89
>> >
>> > 72.13
>> >
>> > -2.399
>> >
>> > < .05*
>> >
>> > L2
>> >
>> > GR vs UNGR
>> >
>> >  -408
>> >
>> >
>> >
>> > 90
>> >
>> > 74.18
>> >
>> > -4.513
>> >
>> > < .001***
>> >
>> > L1
>> >
>> > GR vs UNGR
>> >
>> > -111
>> >
>> >
>> >
>> > 73
>> >
>> > 70.02
>> >
>> > -1.520
>> >
>> > > .05
>> >
>> >
>> >
>> > HL > L2
>> >
>> >  -25
>> >
>> >
>> >
>> > 191
>> >
>> > 43.48
>> >
>> > -.135
>> >
>> > > .05
>> >
>> > GR
>> >
>> > L1 > HL
>> >
>> >  400
>> >
>> >
>> >
>> > 175
>> >
>> > 43.81
>> >
>> > 2.286
>> >
>> > < .05*
>> >
>> >
>> >
>> > L1 > L2
>> >
>> >  374
>> >
>> >
>> >
>> > 179
>> >
>> > 43.59
>> >
>> > 2.092
>> >
>> > < .05*
>> >
>> >
>> >
>> > HL > L2
>> >
>> >  -219
>> >
>> >
>> >
>> > 179
>> >
>> > 42.70
>> >
>> > -1.226
>> >
>> > > .05
>> >
>> > UNGR
>> >
>> > L1 > HL
>> >
>> >  298
>> >
>> >
>> >
>> > 164
>> >
>> > 43
>> >
>> > 1.817
>> >
>> > > .05
>> >
>> >
>> >
>> > L1> L2
>> >
>> > 77
>> >
>> >
>> >
>> > 166
>> >
>> > 42.03
>> >
>> > .469
>> >
>> > > .05
>> >
>> > How would I go about determining the Cohen's *d* for each of the
>> contrasts?
>> >
>> > The model call is:
>> >
>> > Linear mixed model fit by REML. t-tests use Satterthwaite's method
>> > ['lmerModLmerTest']
>> > Formula: RT ~ Group * Grammaticality + (1 + Grammaticality |
>> Participant) +
>> >
>> >     (1 + Group | item)
>> >    Data: RTanalysis
>> >
>> > REML criterion at convergence: 52800
>> >
>> > Scaled residuals:
>> >     Min      1Q  Median      3Q     Max
>> > -2.1696 -0.6536 -0.1654  0.5060  5.0134
>> >
>> > Random effects:
>> >  Groups      Name               Variance Std.Dev. Corr
>> >  item        (Intercept)         71442   267.29
>> >              GroupL2              1144    33.82    0.80
>> >              GroupNS              9951    99.76   -0.43 -0.88
>> >  Participant (Intercept)        235216   484.99
>> >              Grammaticalityungr  50740   225.25   -0.39
>> >  Residual                       378074   614.88
>> > Number of obs: 3342, groups:  item, 144; Participant, 46
>> >
>> > Fixed effects:
>> >                            Estimate Std. Error      df t value Pr(>|t|)
>> > (Intercept)                 2801.98     136.70   48.85  20.498   <2e-16
>> ***
>> > GroupL2                      -25.86     191.20   43.48  -0.135   0.8931
>> > GroupNS                     -400.63     175.22   43.81  -2.286   0.0271
>> *
>> > Grammaticalityungr          -213.87      89.17   72.13  -2.399   0.0190
>> *
>> > GroupL2:Grammaticalityungr  -194.57     107.25   42.55  -1.814   0.0767
>> .
>> > GroupNS:Grammaticalityungr   102.31      99.39   43.45   1.029   0.3090
>> > ---
>> > Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
>> >
>> > Correlation of Fixed Effects:
>> >             (Intr) GropL2 GropNS Grmmtc GrL2:G
>> > GroupL2     -0.672
>> > GroupNS     -0.744  0.526
>> > Grmmtcltyng -0.404  0.222  0.260
>> > GrpL2:Grmmt  0.259 -0.391 -0.205 -0.589
>> > GrpNS:Grmmt  0.299 -0.202 -0.392 -0.702  0.540
>> > convergence code: 0
>> > Model failed to converge with max|grad| = 0.0477764 (tol = 0.002,
>> component
>> > 1)
>> >
>> > The distribution of the outcome is fairly normal and the overall mean,
>> > without considering the two fixed effects, is very close to the means of
>> > each of the three groups (without considering the effect of
>> Grammaticality)
>> > as well as the means of the two levels of grammaticality (without
>> > considering the effect of group).
>> >
>> > The package simR can simulate data to determine power, amongst other
>> things,
>> > but I am not sure how to do this for models with interactions such as
>> mine.
>> >
>> > Use of simR is recommended in Brysbaert and Stevens (2018)
>> > https://www.journalofcognition.org/articles/10.5334/joc.10/. Perhaps
>> there
>> > is a simpler way of extracting *d *from the stats I already know?
>> >
>> > Any help would be greatly appreciated,
>> >
>> > Francesco
>> >
>> >         [[alternative HTML version deleted]]
>> >
>> > _______________________________________________
>> > R-sig-mixed-models using r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>>
> --
> Inviato da Gmail Mobile
>

	[[alternative HTML version deleted]]

More information about the R-sig-mixed-models mailing list