[R-sig-ME] using mixed model vs just using plot means

[Ben Bolker]

On 11-02-22 06:45 PM, Derek Sonderegger wrote:
> I am trying to fully understand the trade-offs between the two analyses and
> if one method is superior in the case of only two observations per random
> effect.
> 
> Full experimental set-up: I have 6 field plots and three of them receive a
> treatment (applied to the plot) and three serve as controls.  In each plot,
> I have measurements from 2 individual plants.  Since this is a desert
> ecosystem, the variance within a plot is quite large compared the the
> variance from plot to plot.
> 
> The 'old-school' analysis method would be to calculate plot average for each
> plot and do an ANOVA using those plot means as data and accept that my
> inference is only at the plot level.  If I go with a fancy mixed model
> approach, my inference is on the level of an individual plant, and I get an
> estimate of within plot variability.
> 
> But the rub is this:  the plot means approach gives me a fairly significant
> treatment effect (p = .014) while the mixed model approach gives me a
> non-significant result (p=.34).  My feeling is that trying to estimate the
> random effect (in addition to the treatment effect) is taking a large amount
> of our statistical power in a relatively data poor problem.  I did a quick
> simulation study and it appears that in the case of high within plot
> variability and a small number of observations per plot, the mixed model
> approach has substantially reduced power compared to the plot means
> 
>>From a scientific point of view, we actually don't mind making inferences at
> the plot level as we are primarily concerned with landscape effects of the
> treatment.
> 
> At this point I think that the plot means approach is the best choice, but I
> want to make sure that I'm not missing anything.
> 

  The plot means approach seems fine to me (see e.g. Murtaugh 2007
Ecology), but I have to admit I'm a little bit concerned by the big
difference in the p values.  In principle (I think) the 'modern' mixed
model approach should converge to the method-of-moments ('old school')
result in the classical case (balanced design, nested effects only,
etc.) ... how are you getting those different p values?  Are you perhaps
doing a likelihood ratio test in one case and an F test in the other?
   I would also say that you must be seeing some pretty big effects to
be getting so much power from what is effectively 6 data points ...

  Ben Bolker