Thank you both-- this is very helpful!

Robert (Bob) Wiley


On Thu, Jun 13, 2013 at 1:41 PM, Jake Westfall <jake987722@hotmail.com>wrote:

> Hi Bob,
>
> 1. For me, the only potential concern I notice is the fact you are going
> with the random-intercept-only version of the model. I think if you are
> able to estimate the full model dictated by the experimental design without
> any convergence issues or obviously crazy estimates and in a reasonable
> amount of time, then it would be best to do this, given that your primary
> statistical goal (I assume) is inference about the fixed effects.
>
> If the variances of the additional random effects turn out to be really
> small, then their inclusion will make little difference for the fixed
> effects anyway, and if they are large, then it is certainly better to
> estimate them than to assume them to be 0.
>
> You might take a look at the recent paper by Barr et al.:
> http://www.sciencedirect.com/science/article/pii/S0749596X12001180
>
> 2. Simple measures of goodness of fit are rather notoriously tricky for
> mixed models. A good place to start for looking into this might be:
>
> http://onlinelibrary.wiley.com/doi/10.1111/j.2041-210x.2012.00261.x/abstract
>
> Jake
>
> > Date: Thu, 13 Jun 2013 09:00:51 -0400
> > From: baron@psych.upenn.edu
> > To: rwwiley@gmail.com
> > CC: r-sig-mixed-models@r-project.org
> > Subject: Re: [R-sig-ME] conceptualizing items + subject analysis
>
> >
> > On 06/12/13 13:37, Bob Wiley wrote:
> > > Hello,
> > >
> > > Let me start by briefly explaining my dataset and the question I'm
> trying
> > > to answer:
> > > Subjects saw pairs of letters and responded either "same" or
> "different"
> > > (e.g. they saw "L L" and responded "same" or "L R" and responded
> > > "different") and I have measures for both accuracy and response time.
> > >
> > > .I want to predict their reaction time on correct "different" responses
> > > based off of several factors:
> > > pixels - the pixel overlap of the two letters
> > > alphabet - their proximity in the alphabet
> > > identity - whether or not they have the same name, like "r R" or "g G"
> > >
> > > Each pair has been coded with a value on all of these measures, for
> example:
> > > "r R" gets a 0.13 on pixels, a 0 on alphabet, and a 1 on identity,
> > > while
> > > "E F" gets 0.64 on pixels, a 1 on alphabet, and a 0 on identity
> > >
> > > So of course I have this large data set of 24*990 observations (less
> > > incorrect responses). But I've gotten confused now about how best to
> > > predict the reaction time on these factors, accounting for the random
> > > effects... I was thinking the R model would be:
> > >
> > > fit = lmer ( rt ~ pixels + alphabet + identity + (1|Subject) +
> (1|Pair))
> > >
> > > The model does run, and in fact all factors come out significant. But I
> > > have two concerns:
> > > 1. That entering the data/defining the model this way is not correct.
> >
> > Why not?
> >
> > The only problem I can see is a substantive one, and I'm not sure
> > about it. Because you are looking at "different" responses only, Ee
> > would be "yes" but EE or ee would be "no". Thus, you have
> > same/different confounded to some extent with
> > same-case/different-case. Similarly, same-case/different-case might be
> > confounded with pixels. Subjects might carry out (simultaneous)
> > comparisons of same-different visually (V), same-different in identity
> > (I), and same-different in case (C). The results of C would determine
> > the relevance of I or V. Thus, you might want to look at what happens
> > if you include a term for C, and terms for its interaction with V and
> > C. (Not sure this is exactly right, but something like this.)
> >
> > Also, I'm not sure how to handle possible individual
> > differences. Subjects may differ in the relative speeds of V, I, and
> > C, and thus in the efficiency of different strategies. If you thought
> > that subjects might reasonably differ in the direction of various
> > effects, then you might want to include random slopes. But if your
> > hypotheses are correctly one-tailed (and I think they are), then I
> > don't see that you need to do this. With data like these, you could
> > also test individual subjects. (See
> > http://www.sas.upenn.edu/~baron/papers/sinica.pdf.)
> >
> > And you might want to include a term for the order of a given trial in
> > the sequence of trials. Subjects get faster throughout an
> > experiment. So including such a term can reduce the variance otherwise
> > attributed to "error" and make the other comparisons more
> > sensitive. In my experience, if you use the log of RT as the dependent
> > variable (usually a good idea anyway, unless you are testing additive
> > factors), then the effect of order is close to linear.
> >
> > > 2. That I need to determine some goodness of fit for the model and
> > > contribution of these factors, the MCMC p-value not being sufficient to
> > > defend the inclusion of all of these variables
> >
> > Here I'm afraid I can't help because I don't understand the
> > problem. There are lots of outputs aside from MCMC p-values. And I
> > don't understand why a low p-value is not sufficient for inclusion.
> >
> > > Any advice or pointing out of my mistakes will be much appreciated.
> Thank
> > > you!
> > >
> > > Bob, JHU
> > >
> > > [[alternative HTML version deleted]]
> > >
> > > _______________________________________________
> > > R-sig-mixed-models@r-project.org mailing list
> > > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
> >
> > --
> > Jonathan Baron, Professor of Psychology, University of Pennsylvania
> > Home page: http://www.sas.upenn.edu/~baron
> > Editor: Judgment and Decision Making (http://journal.sjdm.org)
> >
> > _______________________________________________
> > R-sig-mixed-models@r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>

	[[alternative HTML version deleted]]

