[R-sig-ME] reconstruction of effect sizes in Baayen, Davidson & Bates (2008)

[Dominick Samperi]

On Thu, May 5, 2011 at 7:21 PM, Andrew Robinson
<A.Robinson at ms.unimelb.edu.au> wrote:
> Jan,
>
> surely it would be most appropriate to approach the authors directly?
>
> Andrew
>
> On Thu, May 05, 2011 at 04:24:37PM +0200, VANHOVE Jan wrote:
>> Dear all,
>>
>> I’ve been reading Baayen et al. (2008) for quite some time now in order to get a feel for the usefulness of mixed modeling in language studies. Unfortunately, I can’t wrap my head around the effect sizes that the authors infer from their mixed model outputs. Can anyone put me on the right track? Thanks in advance.

This is a nice survey paper. I copied the data from the paper and tried to
reproduce the results. Everything works as advertised until the 'mcmc$random'
at the end (see below). I get quite different intervals for
item/subject, with a zero lower
bound for item.

Perhaps the differences are due to changes made to lme4 since this
paper was published?

Dominick

## BEGIN CODE
library(lme4)

## Data from Baayen, Davidson, Bates (2008) on mixed-effects modeling.
subj = gl(3,6,labels=c('s1','s2','s3'))
item = factor(rep(c('w1','w2','w3'),6))
SOA  = factor(rep(c('Long','Long','Long','Short','Short','Short'),3))
RT = c(466,520,502,475,494,490,516,566,577,
       491,544,526,484,529,539,470,511,528)

priming <- data.frame(subj=subj,item=item,SOA=SOA,RT=RT)
priming

priming.lmer <- lmer(RT ~ SOA + (1|item) + (1 + SOA|subj),data = priming)
summary(priming.lmer)
fixef(priming.lmer)
ranef(priming.lmer)

## SOA/subj correlation -1, so we try independence:
priming.lmer1 = lmer(RT ~ SOA + (1|item) + (1|subj)+(1|SOA:subj),data=priming)
summary(priming.lmer1)

## Variance of SOA:subj is small, so we try just random intercepts:
priming.lmer2 = lmer(RT ~ SOA + (1|item) + (1|subj),data=priming)
summary(priming.lmer2)

## Compare simplest with most sophisticated model:
anova(priming.lmer, priming.lmer2)

## Simpler model is OK, let's look at it:
priming.lmer2

## Try mcmc
mcmc = mcmcsamp(priming.lmer2,n=50000)
densityplot(mcmc,plot.points=FALSE)

## languageR (linguistics for R by Baayen); uses mcmc from lme4
require(languageR)
mcmc = pvals.fnc(priming.lmer2,nsim=10000)

## HPD intervals for fixed effects
mcmc$fixed

## HPD intervals for random effects
mcmc$random

## END CODE

>>
>> R code and quote re: effect sizes below.
>>
>> R.H. Baayen, D.J. Davidson, D.M. Bates, 2008. Mixed-effects modeling with crossed random effects for subjects and items. Journal of Memory and Language 59, 390-412. Available at http://www.ualberta.ca/~baayen/publications/baayenDavidsonBates.pdf
>>
>> ----begin code----
>> library(languageR)
>> attach(primingHeidPrevRT)
>> # note: RT are log-transformed latencies
>>
>> model1.lmer <- lmer(RT ~ log(RTmin1) + Condition + (1|Word) + (1|Subject))
>> print(model1.lmer, corr=F)
>> # model summary also on p. 400 (top left)
>> ----end code----
>>
>> Quote: ‘The latency to the preceding [i.e. log(RTmin1); JV] has a large effect size with a 400 ms difference between the smallest and the largest predictor values, the corresponding difference for the frequency effect [not in the model; JV] was only 50 ms.’
>>
>> I would like to find out how the authors arrived at these effect size estimates. I can’t reconstruct the 400 ms RTmin1 effect from the model output using the most extreme predictor values:
>>
>> exp((5.80465 + .12125*range(log(RTmin1))))
>> # 685.746 820.627 --- i.e. a 135 ms difference
>>
>> Incidentally, this 135 ms difference corresponds to the partial effect I can glean from the plot produced by plotLMER.fnc():
>>
>> plotLMER.fnc(model1.lmer, pred="log(RTmin1)")
>>
>> Similarly, I’m at a loss as to how they arrived at the 50 ms frequency effect as frequency (in contrast to BaseFrequency) does not seem to be coded in the data set as well as at the 100 ms ‘ResponseToPrime’ effect in another model (p. 400, bottom left).
>>
>> Thanks,
>> Jan
>>
>> ---
>> Jan Vanhove
>> jan.vanhove at unifr.ch
>>
>> PhD student
>> Section Multilingualism and foreign language didactics
>> University of Fribourg, Switzerland
>>
>> _______________________________________________
>> R-sig-mixed-models at r-project.org mailing list
>> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>
> --
> Andrew Robinson
> Program Manager, ACERA
> Department of Mathematics and Statistics            Tel: +61-3-8344-6410
> University of Melbourne, VIC 3010 Australia               (prefer email)
> http://www.ms.unimelb.edu.au/~andrewpr              Fax: +61-3-8344-4599
> http://www.acera.unimelb.edu.au/
>
> Forest Analytics with R (Springer, 2011)
> http://www.ms.unimelb.edu.au/FAwR/
> Introduction to Scientific Programming and Simulation using R (CRC, 2009):
> http://www.ms.unimelb.edu.au/spuRs/
>
> _______________________________________________
> R-sig-mixed-models at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>