[R-sig-ME] Help with determining effect sizes
João Veríssimo
j|@ver|@@|mo @end|ng |rom gm@||@com
Sat Oct 5 13:08:09 CEST 2019
See the web app by Jake Westfall:
https://jakewestfall.shinyapps.io/crossedpower/
And their JEP:General paper:
http://doi.org/10.1037/xge0000014
If I'm not mistaken, you would standardise the estimates of differences
by the sum of all variances (random intercepts and slopes + residual),
but you'll need to make sure that's the right formula (given your
desgin).
João
On Sat, 2019-10-05 at 12:13 +0200, Maarten Jung wrote:
> Dear Francesco,
>
> I don't think there is a "standard" way to calculate effect sizes for
> linear mixed models due to the way the variance is partitioned (see
> e.g. [1]).
> One way to compute something similar to Cohen's d would be to divide
> the difference between the estimated means of two conditions by a
> rough estimate of the standard deviation of the response variable
> which you can get by
> sd(predict(your_model_name))
>
> Best,
> Maarten
>
> [1]
> https://afex.singmann.science/forums/topic/compute-effect-sizes-for-mixed-objects#post-295
>
>
> On Sat, Oct 5, 2019 at 10:01 AM Francesco Romano <
> fbromano77 using gmail.com> wrote:
> >
> > Dear all,
> >
> > A journal has asked that I determine the effect sizes for a series
> > of
> > dummy-coded contrasts from the following ME model:
> >
> > RT ~ Group * Grammaticality + (1 + Grammaticality | Participant) +
> > (1 + Group | item)
> >
> > Here RT is my continuous outcome variable measured in milliseconds,
> > Group
> > is a factor with 3 levels (NS, L2, and HL), and Grammaticality a
> > factor
> > with 2 levels (gr and ungr). After relevelling —NOTE: I am
> > deliberately
> > omitting the call for each new relevelled model here— I obtained a
> > series
> > of contrasts which are tabulated below (not sure you can view this
> > whole):
> >
> >
> > Reference level
> >
> > Contrasts
> >
> > Estimate
> >
> > (ms)
> >
> > Effect size
> >
> > (Cohen’s *d*)
> >
> > SE
> >
> > df
> >
> > *t*
> >
> > *p*
> >
> > HL
> >
> > GR vs UNGR
> >
> > -213
> >
> >
> >
> > 89
> >
> > 72.13
> >
> > -2.399
> >
> > < .05*
> >
> > L2
> >
> > GR vs UNGR
> >
> > -408
> >
> >
> >
> > 90
> >
> > 74.18
> >
> > -4.513
> >
> > < .001***
> >
> > L1
> >
> > GR vs UNGR
> >
> > -111
> >
> >
> >
> > 73
> >
> > 70.02
> >
> > -1.520
> >
> > > .05
> >
> >
> >
> > HL > L2
> >
> > -25
> >
> >
> >
> > 191
> >
> > 43.48
> >
> > -.135
> >
> > > .05
> >
> > GR
> >
> > L1 > HL
> >
> > 400
> >
> >
> >
> > 175
> >
> > 43.81
> >
> > 2.286
> >
> > < .05*
> >
> >
> >
> > L1 > L2
> >
> > 374
> >
> >
> >
> > 179
> >
> > 43.59
> >
> > 2.092
> >
> > < .05*
> >
> >
> >
> > HL > L2
> >
> > -219
> >
> >
> >
> > 179
> >
> > 42.70
> >
> > -1.226
> >
> > > .05
> >
> > UNGR
> >
> > L1 > HL
> >
> > 298
> >
> >
> >
> > 164
> >
> > 43
> >
> > 1.817
> >
> > > .05
> >
> >
> >
> > L1> L2
> >
> > 77
> >
> >
> >
> > 166
> >
> > 42.03
> >
> > .469
> >
> > > .05
> >
> > How would I go about determining the Cohen's *d* for each of the
> > contrasts?
> >
> > The model call is:
> >
> > Linear mixed model fit by REML. t-tests use Satterthwaite's method
> > ['lmerModLmerTest']
> > Formula: RT ~ Group * Grammaticality + (1 + Grammaticality |
> > Participant) +
> >
> > (1 + Group | item)
> > Data: RTanalysis
> >
> > REML criterion at convergence: 52800
> >
> > Scaled residuals:
> > Min 1Q Median 3Q Max
> > -2.1696 -0.6536 -0.1654 0.5060 5.0134
> >
> > Random effects:
> > Groups Name Variance Std.Dev. Corr
> > item (Intercept) 71442 267.29
> > GroupL2 1144 33.82 0.80
> > GroupNS 9951 99.76 -0.43 -0.88
> > Participant (Intercept) 235216 484.99
> > Grammaticalityungr 50740 225.25 -0.39
> > Residual 378074 614.88
> > Number of obs: 3342, groups: item, 144; Participant, 46
> >
> > Fixed effects:
> > Estimate Std. Error df t value
> > Pr(>|t|)
> > (Intercept) 2801.98 136.70 48.85 20.498 <2
> > e-16 ***
> > GroupL2 -25.86 191.20 43.48 -
> > 0.135 0.8931
> > GroupNS -400.63 175.22 43.81 -
> > 2.286 0.0271 *
> > Grammaticalityungr -213.87 89.17 72.13 -
> > 2.399 0.0190 *
> > GroupL2:Grammaticalityungr -194.57 107.25 42.55 -
> > 1.814 0.0767 .
> > GroupNS:Grammaticalityungr 102.31 99.39 43.45 1.029 0.
> > 3090
> > ---
> > Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
> >
> > Correlation of Fixed Effects:
> > (Intr) GropL2 GropNS Grmmtc GrL2:G
> > GroupL2 -0.672
> > GroupNS -0.744 0.526
> > Grmmtcltyng -0.404 0.222 0.260
> > GrpL2:Grmmt 0.259 -0.391 -0.205 -0.589
> > GrpNS:Grmmt 0.299 -0.202 -0.392 -0.702 0.540
> > convergence code: 0
> > Model failed to converge with max|grad| = 0.0477764 (tol = 0.002,
> > component
> > 1)
> >
> > The distribution of the outcome is fairly normal and the overall
> > mean,
> > without considering the two fixed effects, is very close to the
> > means of
> > each of the three groups (without considering the effect of
> > Grammaticality)
> > as well as the means of the two levels of grammaticality (without
> > considering the effect of group).
> >
> > The package simR can simulate data to determine power, amongst
> > other things,
> > but I am not sure how to do this for models with interactions such
> > as mine.
> >
> > Use of simR is recommended in Brysbaert and Stevens (2018)
> > https://www.journalofcognition.org/articles/10.5334/joc.10/.
> > Perhaps there
> > is a simpler way of extracting *d *from the stats I already know?
> >
> > Any help would be greatly appreciated,
> >
> > Francesco
> >
> > [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-mixed-models using r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
> _______________________________________________
> R-sig-mixed-models using r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
More information about the R-sig-mixed-models
mailing list