Hi:

This is a very nice precis, Robert, and a welcome contribution. I'd just
like to add a couple of comments in
the hope that it adds a little to this discussion.

On Fri, Apr 23, 2010 at 9:13 AM, Robert A LaBudde <ral@lcfltd.com> wrote:

> The question isn't whether an effect is "random" or "fixed". Anything can
> be considered either under different conditions. The real question is one of
> inference: Are you willing to make statements only about "these" blocks, or
> do you want to make inferences about "those" blocks too.
>
> Consider the simplest case: a randomized block design. Suppose you have 3
> blocks upon which you apply the treatments you want to measure. The
> treatments are applied in replicate within blocks.
>
> If you treat the blocks as a fixed effect, you are saying you only care
> about inferences of treatment effect on "these" blocks, but no others. The
> blocks are then just an artifact of the design used to reduce variance of
> the treatment effect.
>
> If the treat the blocks as a random effect, you are saying you want your
> inferences on treatment effect to apply not only to "these" blocks, but also
> hypothetically all "those" other blocks of similar type. You would also then
> be interested in the standard deviation of the block effect of that
> population of blocks.
>

This is referred to as the 'scope of inference', discussed to some degree in
the SAS for Mixed Models manual. It's an important
consideration in designs/models with random and fixed effects.

>
> Now we get to the practical side. You can treat the blocks as random with
> only 3 blocks, but you will get a very imprecise estimate of the block
> variance. This is not doing something wrong, it's only a fact of estimation.
> You will still control for block in measuring treatment effect. The bad news
> is that the imprecision in block variance will lead to imprecision in
> treatment effect. What then do you want to do? Treat blocks correctly as
> random, but get imprecise estimates? Or treat blocks as fixed, but get a
>  precise treatment estimate, but with a confidence interval that's a little
> too liberal?
>

This is an excellent and often overlooked point. It's also a design issue:
if you really want to treat blocks as fixed, then you want to
replicate them sufficiently to achieve decent power/coverage for the
tests/CIs. OTOH, if you want to treat blocks as random, then it's
better to trade off replication for more levels to get a better estimate of
the block variance. IIRC, this is related to the Neyman-Scott problem...
So in a practical sense, and I think this gets at Ben's question a bit,
these types of decisions need to be made prior to data collection and
addressed as part of the design.

To quote a line Steve Vardeman made once in a class I took with him:
"There's no free lunch in statistics".  It seems especially apropos in this
discussion: if you want to treat a three-level factor as random and estimate
its variance component from those three levels, it's very likely to be
imprecisely estimated, and that imprecision may transmit elsewhere in the
analysis; it you'd rather treat it as fixed, you lose the ability to
generalize
beyond the three observed levels. Gabor's experiment shows how much
imprecision results from trying to estimate a variance component
with very few levels and how it can be improved with more observed levels.


> Personally, I'm a purist. If blocks are random, I want to model them as
> random, even if I've only got 3 or 4 of them. I rather have a correct model
> (low bias) than precise, biased estimates. Others have different opinions.
> Hence the 6 blocks minimum rule of thumb.
>
> Note that any effect can be theoretically fixed or random. Consider a 1-way
> ANOVA with 3 drugs tested in replicate (mice, say). Normally this is treated
> as a fixed effect model, because we only care about the effects of "these"
> three drugs. But suppose I hypothecate the 3 drugs were chosen at random
> from some sampling frame of drugs, and I want to make inferences about
> variability of treatment effect within "that" population of drugs. Then  I
> might use a random effect model to get a standard deviation of treatment
> effect.
>

'hypothecate'??  :)

>
> The "elephant in the room" on this question is that rarely do people
> actually have random sampling behind the choice of the random effect
> subjects/blocks/centers. That introduces considerable selection bias and
> confusion about what population is really being represented. If your blocks
> aren't being randomly selected or at least representative of the population
> of inference, that is a more important issue than whether you model the
> effect as fixed or random. So this is more justification for the ad hoc 6
> blocks minimum rule, and creates more acceptability for the fixed effect
> model.
>

I've found this to be true in practice, too, and it goes to.....design (and
its execution).

Re Ben's points,:
     1. philosophy, schmilosophy.  Fit these factors as a fixed effect,
         anything else is too dangerous/misleading/unworkable.

I agree with Robert in principle re purism, but the reality of the situation
can't be ignored
either; once the data are in hand, design problems are moot (unless you're
in the
rare position where the design can be repaired). To me the issue is:
which trade-off (precision of estimation or appropriate scope of inference)
is
more important for this problem? This is 'no free lunch' in action...


    2. proceed with the 'standard' mixed model (lme4, nlme, PROC MIXED,
        ...) and hope it doesn't break.  Ignore warnings.

Warnings are ignored at one's own peril; they exist for a reason, and I
would
hope that they would be taken seriously. But once again, there are practical

trade-offs that may need to be made for the situation at hand.

 3. use Bayesian-computational approaches (MCMCglmm, WinBUGS, AD Model
Builder with post-hoc MCMC calculation? Data cloning?)?  Possibly with
half-Cauchy priors on variance as recommended by Gelman [Bayesian
Analysis (2006) 1, Number 3, pp. 515–533]?

To a degree, this question is sort of like asking which type of cosmetic
surgery
should be performed on the data once its flaws have been exposed. What
matters
is the types of inferences required and whether the assumptions that
underlie them are understood by, accepted by and most importantly, verified
by,
the analyst/investigator. Bayesian methods certainly have their place in
mixed
model methodology, and if they answer the investigator's questions, great.
But
the scope of inference of a credible interval arising from a posterior
distribution
is to the study at hand (local); if the investigator is expecting
generalization to a
wider population, that may be a problem...and 'no free lunch' strikes again.

HTH,
Dennis

	[[alternative HTML version deleted]]


