Reinhold,

The maximum possible specification for the AMP design would be, in lme4 syntax:

y ~  1+C + (1+C|S) + (1+C|T) + (1|P) + (1|S:P) + (1+C|S:T) + (1|P:T) + (1|S:P:T)

For all of these random interaction terms to be unconfounded from the residuals and thus estimable, we need each Subject to respond to every pairing of Prime and Target multiple times. Note, actual implementations of the paradigm this example is based on would not typically incorporate enough trials for us to get reasonable estimates of all of these variance components, but this is a hypothetical example, and they are possible in principle to estimate.

Jake

Date: Sat, 9 Nov 2013 17:18:41 +0100
Subject: Re: [R-sig-ME] Variances of higher-order vs. lower-order random terms
From: reinhold.kliegl@gmail.com
To: jake987722@hotmail.com
CC: r-sig-mixed-models@r-project.org

I am not quite sure I understand how you specify your models. Perhaps my question is too specific for the example you gave, but would you not specify S, P, and T as crossed-random factors, that is assume that the random factors are independent? For example, using a numeric sum contrast for C and assuming that condition is a between-P factor, I would specify experiment 1 as follows:
  y ~  1 + C + (1+C | S) + (1+C | T) + (1 | P)
This model yields 5 variance components and 2 correlation parameters (in addition to the residual variance and fixed effects of intercept and C). What does your specification look like?



On Fri, Nov 8, 2013 at 9:42 PM, Jake Westfall <jake987722@hotmail.com> wrote:

Thanks Reinhold, this is a great point.  I think the issue you raise would give us good reason to be skeptical of the hierarchical ordering principle in the case of multiple fixed factors. I can certainly think of common experimental paradigms in psych/neuroscience where the effects of primary interest are interaction effects (e.g., many priming tasks which examine congruency/incongruency differences). But I think the principle would still hold for experiments containing only a single fixed factor, and for the non-interchangeable random factors (and interactions thereof) in an experiment that did contain multiple fixed factors. See if you agree with my reasoning below for each of these two cases.




Consider first the case of a balanced experiment where we have multiple random factors, but only one fixed factor. To use an example from my own field, in social psych we have the "affect misattribution task" which is basically a modified priming task where the target stimuli are all ambiguous Chinese ideographs (ambiguous to American participants), but there are two prime types or conditions. So we have 3 random factors, Subject, Prime, Target, and 1 fixed factor, Condition. Primes are nested in Condition, so the experiment has 11 estimable effects: 4 main effects (S, P, T, C), 5 two-way interactions (S:P, S:T, P:T, S:C, T:C), and 2 three-way interactions (S:P:T, S:T:C), and the only fixed effect out of all of these is the main effect of Condition.




In this kind of experiment, with only a single fixed factor, we cannot manipulate the codings to interchange main effects and interactions among the fixed effects, because there is only one and so nothing to interchange with. And certainly it wouldn't make sense to try and interchange fixed and random factors in this way. So my claim is that the hierarchical ordering principle would tend to apply here, such that the variances of the random S, P, T effects will tend to be greater than the variances of the S:P, S:T, P:T, S:C, T:C effects, which in turn will tend to be greater than the variances of S:P:T, S:T:C effects.




Next consider the case of a more traditional priming experiment, similar to the A.M.P. described above, but we additionally there are two types of Targets. And the effect of primary interest here might be the fixed Prime type * Target type interaction, which perhaps represents a congruent vs. incongruent difference. As you note, we could manipulate the coding of our 4 cells here so that congruent vs. incongruent was one of the main effects and not an interaction. This makes general claims about the variances of effects related to the simple and interaction effects dubious.




But there is still a hierarchy of main and interaction effects just among the random factors in the experiment. In this case we have the random effects of S, P, T; the two-way interactions of S:P, S:T, P:T; and the three way interaction S:P:T. So while we might be skeptical about applying the hierarchical ordering principle to the fixed factors and to random effects that involve the fixed factors (i.e., random slopes), I claim that hierarchical ordering would still tend to apply to the subsets of random effects in the experiment that I listed above.




What do you think about these two cases?



Jake



Date: Fri, 8 Nov 2013 19:49:50 +0100

Subject: Re: [R-sig-ME] Variances of higher-order vs. lower-order random terms

From: reinhold.kliegl@gmail.com

To: jake987722@hotmail.com

CC: r-sig-mixed-models@r-project.org



If I understand your question correctly, I think there is one exception. Whenever you are dealing with balanced 2 x 2 x 2 x ... design all main effects and interactions have equal statistical power. Whether or not main effects are more important than interactions depends only on the coding of the factors, that is you can recode any interaction as a main effect (e.g., congruency effects are often interactions). Of course, in a typical research program main effects usually represent large known effects, but this is a conceptual reason, not a mathematical implication. Therefore, in my opinion, if a large effect maps onto a higher-order interaction, it may well be associated with larger effects (and variance components) than main effects.  The first paper I know that discusses this issue is:


Shaffer, J.P. (1977). Reorganization of variables in analysis of variance and multidimensional contingency tables. Psychological Bulletin, 84, 220-228. I stumbled upon this when I realized that a significant 2 x 2 x 2 interaction mapped onto a simple main effect in a psycholinguistic experimental design.




I am not sure whether this case generalizes to factorial designs with factors with more than 2 levels, but in LMMs we usually test single-degree-of-freedom contrasts. So, perhaps as long as the contrasts are orthogonal it may hold here as well. (Shravan Vasishth recently expressed some doubts about this, but we have not sorted this out completely--I think.)




When you are dealing with continuous covariates (non-uniformly, usually uni-modally distributed variables), interactions terms have definitely less power. My classic reference for this is McClelland, G.H., & Judd, C.M. (1993). Statistical difficulties of detecting interactions and moderator effects. Psychological Bulletin, 114, 376-390. So in this case, I think the low statistical power will lead to small effects and little chance to detect significant variance components for interactions involving such variables.




Reinhold

























On Fri, Nov 8, 2013 at 7:15 PM, Jake Westfall <jake987722@hotmail.com> wrote:



Hi all,



I wanted to follow up with a tentative answer to the question I posted to the list a few weeks ago concerning lower-order random effects tending to have greater variance than higher-order random effects. There were not a lot of takers to my question either here or on stats.stackexchange.com, which is not too surprising really since it was kind of a long, involved, and ambiguous question.










I have discovered that the regularity I described in my question has in fact been written about by several authors in the literature on Design of Experiments (DoE). It has been called the "hierarchical ordering principle" and also sometimes the "sparsity-of-effects principle."










In the chapter on fractional factorial designs in Montgomery (2013, p. 290), he writes:



>The successful use of fractional factorial designs is based on three key ideas:



>



>1. The sparsity of effects principle. When there are several variables, the system or process is likely to be driven primarily by some of the main effects and low-order interactions.



>



>...











Wu & Hamada (2000, p. 143) instead call this the "hierarchical ordering principle", and use the phrase "sparsity of effects" to refer to a related but distinct observation:



>Three fundamental principles for factorial effects:



>



>Hierarchical ordering principle: (i) Lower order effects are more likely to be important than higher order effects, (ii) effects of the same order are likely to be equally important.



>



>Effect sparsity principle: The number of relatively important effects in a factorial experiment is small.



>



>...















 - Montgomery, D. C. (2013). Design and analysis of experiments (Vol. 8). New York: Wiley.



 - Wu, C. J., & Hamada, M. S. (2000). Experiments: planning, analysis, and optimization (Vol. 552). John Wiley & Sons.







Jake



From: jake987722@hotmail.com



To: r-sig-mixed-models@r-project.org



Subject: RE: Variances of higher-order vs. lower-order random terms



Date: Tue, 15 Oct 2013 02:25:29 -0600



















Follow-up: I cross-posted this question here:







http://stats.stackexchange.com/q/72819/5829







Jake







From: jake987722@hotmail.com



To: r-sig-mixed-models@r-project.org



Subject: Variances of higher-order vs. lower-order random terms



Date: Tue, 15 Oct 2013 02:08:58 -0600



















Hi everyone,







**TL, DR summary:**



Is there any theoretical or empirical basis to support the following statement being true as a general rule of thumb?



"When estimating a mixed model, typically the estimated variances/standard deviations of random effects associated with 'higher-order' terms (e.g., random effects of two-way, three-way, and beyond interaction terms) turn out to be *smaller* than the estimated variances/standard deviations of random effects associated with 'lower-order' terms (e.g., the residual variance, variances associated with simple effects of grouping factors)."










The source of this claim is me. ;)







****







Okay, now for the longer version...







Typically when I sit down to start analyzing a new dataset which I know will call for a mixed model, one of the first models that I fit (after the statistical foreplay of looking through the observations in the dataset, plotting various things, cross-tabulating different factors, etc.) is one that is pretty close to the "maximal" random effects specification, where every random effect that is in-principle possible to estimate from the data, is estimated.










Naturally, it is not uncommon that this nearly-maximal model will have some computational problems (convergence errors, or wacky variance/covariance estimates, or etc.) and that I have to trim back this model to find one that my data can more easily support. Fine.










In these situations, the method I have come to prefer for trimming random terms is not to rely on significance tests or likelihood ratios, but rather to just identify the random effects that seem to have the smallest standard deviations (which can admittedly be a little tricky when predictors are on very different scales, but I try to take account of this in my appraisal) and remove these terms first, sequentially in an iterative process. The idea being that I want to alter the predictions of the model as little as possible while still reducing the complexity of the model.










One pattern that I seem to have noticed after a pretty good amount of time spent doing this is that following this method very often leads me to trim random effects associated with higher-order terms (as defined above) of the model first. This is not always true, and occasionally some of the higher-order terms explain a lot of variance, but this doesn't seem to be the general pattern. In sharp contrast, I usually find that lower-order random terms -- particularly those associated with simple effects of the grouping factors -- explain a pretty good amount of variance and are fairly essential to the model. At the extreme, the residual term commonly accounts for close to the most variance, although of course removing this term wouldn't be sensible.










This entirely informal observation leads me to form the hypothesis that I stated at the beginning of this email.







If it is true, then it constitutes a useful piece of advice that might be passed down to people who are less experience with this kind of model selection process. But before I begin doing so, I want to check with other, more experienced users about their reactions to this observation. Does it seem more or less true to you? Is it roughly consistent with your experience fitting many different mixed models to many different datasets? Do you know of any sensible, theoretical reasons why we might actually *expect* this to be true in a lot of cases? Or does it just seem like bullshit?










One possible answer here is that it is not true even in my own case, and I have simply deceived myself. Certainly a possibility that I am open to.







Another possibility is that it might be true in my own case, but that this could simply be a kind of coincidence having to do with the kinds of datasets that I tend to work with routinely (which, FYI, are datasets in psychological / social sciences, a slight majority being experimental in origin, but also a fair proportion of non-experimental stuff). If this is the case then there is probably no good reason for expecting my observations to hold in general in other fields that handle very different kinds of data. Still, if there is a coherent non-coincidental reason for why this might be expected to be true, even if only for these particular kinds of datasets, I would love to hear it.










And of course another possibility is that others *have* noticed similar patterns in their own data, and that it represents some kind of general rule of thumb that people find useful to keep in mind as they fit mixed models to various different data. If this is the case then it seems like there must be some compelling statistical-theoretical reason for why this pattern arises. But I really don't know what that reason would look like.










I welcome anyone's thoughts and opinions about this. Totally legitimate responses to this might be as simple as comments like "Yeah I have noticed something similar in the data I've worked with, but I have no idea why it should be true" or conversely "I have noticed nothing like this in the data I've worked with." Of course I also welcome longer and more involved discussions...










FULL DISCLOSURE: I think I am probably also going to post this question to stats.stackexchange.com. If/when I do, I will send along the link to that question thread.










Jake















        [[alternative HTML version deleted]]







_______________________________________________



R-sig-mixed-models@r-project.org mailing list



https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models







        [[alternative HTML version deleted]]



_______________________________________________

R-sig-mixed-models@r-project.org mailing list

https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models


 		 	   		  
	[[alternative HTML version deleted]]

