Indeed, if one were to fit the same model in Stan, we would see that the
posterior distribution of the correlations is dominated by the priors
(i.e., near 0) and has huge uncertainty (this dominance by the prior goes
down as number of items and number of subjects is increased, as one would
expect).

Example code is in the linear mixed models chapter in lecture notes from a
course I teach at Potsdam:

http://www.ling.uni-potsdam.de/~vasishth/bdalecturenotes.pdf

Regarding the likelihood ratio test, yes, that makes a lot of sense. But
maybe the real problem is the obsession with p<0.05 rather than with
replications. I saw recently that it's a policy in the journal Cognition
that p<0.05 should be considered significant, but p<0.06 not:

http://www.elsevier.com/journals/cognition/0010-0277/guide-for-authors

and

http://ees.elsevier.com/cognit/img/Editorial_Policy.pdf

"However, unless converging evidence is presented, effects that
only approach this level (e.g. at p<.06 or even p<.1) should be interpreted
with considerable caution. Cognition
will require its authors to adhere to the convention that an effect
described as 'statistically significant' must
have a p-value below .05 (for better or for worse, this is the current
convention)."

To be fair to Cognition, they also say: "However, unless converging
evidence is presented, effects that
only approach this level (e.g. at p<.06 or even p<.1) should be interpreted
with considerable caution." But I somehow doubt that people will realize
that this is the more important part of the message.

Instead of recommendations like these, if we were to insist on (a) making
sure we run high power studies (e.g., by computing n in advance for an
expected size of effect), (b) demonstrable replicability of the pattern,
many of these discussions about whether to fit a maximal model or not would
disappear. It's because we sometimes find ourselves scraping of the bottom
of the barrel  because of policies like the above that we end up in
situations like the present one. (I know Roger knows all this, I just
wanted to put it out there).

best,


--
Shravan Vasishth
Professor for Psycholinguistics and Neurolinguistics
Department of Linguistics
University of Potsdam, Germany
http://www.ling.uni-potsdam.de/~vasishth


On Fri, Mar 7, 2014 at 11:13 AM, Levy, Roger <rlevy@ucsd.edu> wrote:

>
> On Mar 7, 2014, at 6:21 AM, Shravan Vasishth <vasishth.shravan@gmail.com>
> wrote:
>
> > Hi Roger and Emilia, and others,
> >
> > I just wanted to say that in Emilia's data, she has 36 subjects and 20
> > items. Roger, would you agree that it is very difficult with this amount
> of
> > data to accurately estimate the full variance-covariance matrices for
> > subjects and for items random effects, especially the correlation
> > parameters? The numbers that lmer returns, for such sizes of data, are
> > pretty wild estimates, and often have no bearing to the true underlying
> > correlations. I think that in this situation we might be asking too much
> > from lmer, without giving it enough data. If, on the other hand, we have
> a
> > lot of data by subjects and items, it becomes possible to estimate these
> > parameters.
> >
> > I believe this may have been, at least partly, the intent of Douglas
> Bates'
> > original message about overparameterization.
>
> That's a good question.  I imagine there is a fair bit of uncertainty
> regarding the correlation parameters, though I would guess that it's not
> huge for this-sized dataset. The point estimates that lme4(.0) give us
> don't quantify this uncertainty, but of course we could use Bayesian
> methods to get a better sense of them.
>
> More generally, this point that you raise, Shravan, is precisely the
> reason that I tend to favor likelihood-ratio tests over the t-statistic for
> the purposes of confirmatory hypothesis tests like Emilia's.  As Baayen,
> Davidson and Bates (2008, page 396) crucially point out, the t-statistic is
> computed conditional on a point estimate of the random-effects covariance
> matrix, and fails to take into account uncertainty in the estimate of this
> matrix.  The likelihood ratio does not have this problem.  (It has other
> problems -- namely that the log likelihood ratio is not truly chi-squared
> distributed -- but with 20 items and 36 subjects in a balanced design I
> would expect that the chi-squared approximation is fairly close.  And at
> any rate, the same problem exists with the t statistic.)
>
> So my take is that how much we should worry about these issues depends in
> part on our modeling goals.  For a confirmatory hypothesis test like
> Emilia's on her dataset, I wouldn't worry much about overparameterization
> for the models she was showing us.  If she wanted to aggressively interpret
> the parameter estimates resulting from a particular model fit, on the other
> hand, I would be much more cautious.
>
> Best
>
> Roger

	[[alternative HTML version deleted]]

