Hi:

On Fri, Jan 7, 2011 at 7:17 PM, Steven McKinney <smckinney@bccrc.ca> wrote:

>
> Were you collecting this data so that you could compare decay rates over
> time?
> If so, then this test would be a test of an a-priori specified hypothesis.
>
> Under the null hypothesis, all lots have the same slope.
> Under the alternative hypothesis, all lots have different slopes.
>

Steven, your description of the alternative hypothesis is not quite correct
- it should be that not all lots have the same slope, i.e., at least two
slopes are different. It is possible that all lots have different slopes,
but there are many other possible simple alternatives in the composite
hypothesis as well. [I'm pretty certain you know that.] Basically, a correct
statement of the alternative is 'not H_0'.


> So you would have one omnibus test comparing the 21 slopes for any
> difference.
> The null model would have one parameter for slope, for all lots.
> The alternative model would have 21 parameters for slope, one for each lot.
>
> I'm not sure of the necessity for a mixed effects model - the little data
> shown here suggests you could just do a linear model exercise.  You state
> that the lots are independent.
>

The central question in this regard pertains to the scope of inference: are
the 21 lots in question the only ones of interest, or can they reasonably be
considered a random sample from a larger population of such lots? If the
latter situation is in force, there are (fortunately) enough lots to well
estimate its variance component.


>
> The single omnibus test will yield a result that protects against multiple
> comparisons.
>

This is certainly true for the Fisher LSD procedure, but I think it's a bit
of an overstatement in general. The statistical properties of many MC
procedures are not conditional on the result of the omnibus F test. I agree
with the general tenor of the remarks below, however.

Done as a linear model exercise, the test statistic would be an F statistic
> with
> 20 df in the numerator.
>

> If this test result is not significant, then the one apparently
> different-looking
> lot is just a result of random variation seen across this much data.
>
> If this test result is significant, then the next question is which lots
> differ
> from which others?  Multiple comparison procedures can then be used to
> illustrate that issue.
>
> It would be data-snooping to pick out the one obviously different lot, and
> start
> comparing it to the other 20.  Test statistic p-values would then be too
> liberal
> and confidence intervals too narrow.
>

This is an important point. For example, if the inclination a priori (i.e.,
in the design phase of the study) was to perform all pairwise comparisons by
the Tukey-Kramer method (just play along :), and then you changed your mind
after seeing one lot deviate obviously from the others to test something
closer to a Dunnett procedure (i.e., comparing all treatments against the
deviant lot, but not necessarily with the deviant lot acting as a
control...more like a Bonferroni procedure), that would not be kosher
because you used the data to modify the hypothesis set in such a way that
was favorable to the outcome you observed. How so? Instead of the 21 C 2 =
210 pairwise tests from T-K, you would be testing 20 instead, artificially
increasing the residual df for the test. This makes the comparisonwise tests
more sensitive, increasing the power of the test procedure, hence improving
the chances of rejecting the null hypothesis of 'no lot mean differences'
[relative to T-K]. This is the statistical analogue of changing horses in
midstream.

For the purposes of testing, you kind of have to turn a blind eye to what
was observed and not let it influence what you would have tested had you not
observed it.

If lots are supposed to be a random factor, then the issue of how to deal
with the deviant lot is rather interesting. Clearly, its inclusion will
(artificially?) increase the size of the estimated variance component, so
the question that needs to be considered is whether the 'rogue' lot is a
plausible representative of the population of all lots of that chemical or
whether there is something unique about it that makes it
'non-representative'- e.g., that it comes from a different supplier than the
other lots in the sample. One approach to modeling could be to estimate
mixed models with and without the lot in question and (i) observe the
difference between variance component estimates, (ii) note how much impact
the inclusion/exclusion of the lot in question impacts the statistical
conclusions of the study. If there are substantial differences in
statistical conclusions, then the disposition of that lot is a critical
question and interest should focus on its resolution. If the conclusions are
essentially the same in both cases, then the unusual nature of the 'rogue'
lot is interesting but not a difference maker. More likely, the results are
somewhere in between, which is why they pay you the big bucks as a
statistician :)

HTH,
Dennis

>
> lmf <- lm(CHG ~ Days * Lot, data = Conc.dat) # Each lot has different slope
> and intercept
> lmr <- lm(CHG ~ Days + Lot, data = Conc.dat) # Each lot has same slope but
> different intercept
> anova(lmr, lmf)
>
> should give you the omnibus F test.
>
> Check data plots,  residual plots and other diagnostics for any
> pathological issues in the model fits.
>

>
> Steven McKinney, Ph.D.
>
> Statistician
> Molecular Oncology and Breast Cancer Program
> British Columbia Cancer Research Centre
>
> email: smckinney +at+ bccrc +dot+ ca
>
> tel: 604-675-8000 x7561
>
> BCCRC
> Molecular Oncology
> 675 West 10th Ave, Floor 4
> Vancouver B.C.
> V5Z 1L3
> Canada
>
> ________________________________________
> From: r-sig-mixed-models-bounces@r-project.org [
> r-sig-mixed-models-bounces@r-project.org] On Behalf Of Prew, Paul [
> Paul.Prew@ecolab.com]
> Sent: January 7, 2011 1:27 PM
> To: r-sig-mixed-models@r-project.org
> Subject: [R-sig-ME] Data snooping in repeated measures study?
>
> Hello, I've been asked to perform an analysis that I'm not sure how to
> frame properly, or even if it can be performed validly.
>
> There are 21 independent batches ("Lots") of a chemical that were measured
> repeatedly over a number of months.  The response measured was the
> concentration of the active ingredient "CHG", with interest in how the CHG
> decays over time.  One Lot had a slope much lower than the other Lots.  Is
> it possible to test the slope of this Lot for statistical significance, with
> the null hypothesis that the slope is no different than the overall slope
> for this chemical?  Or would any test constructed just be data snooping,
> invalidating any inference?  Can anyone suggest a valid approach using lme?
>
> > str(Conc.dat)
> 'data.frame':   121 obs. of  3 variables:
>  $ Lot : Factor w/ 21 levels "L012391","L012471",..: 16 16 16 16 16 16 16
> 16 10 10 ...
>  $ Days: int  30 121 217 307 399 583 765 766 78 176 ...
>  $ CHG.: num  2.06 2.01 1.97 1.94 1.88 …
>
> All of the Lots were measured over 2 years time, but have different numbers
> of intermediate measurements.
>
> Thank you, Paul
>
> Paul Prew   ▪  Statistician
> 651-795-5942   ▪   fax 651-204-7504
> Ecolab Research Center   ▪  Mail Stop ESC-F4412-A
> 655 Lone Oak Drive   ▪   Eagan, MN 55121-1560
>
>
>
> CONFIDENTIALITY NOTICE:
> This e-mail communication and any attachments may contain proprietary and
> privileged information for the use of the designated recipients named above.
> Any unauthorized review, use, disclosure or distribution is prohibited.
> If you are not the intended recipient, please contact the sender by reply
> e-mail and destroy all copies of the original message.
>
>
>        [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-mixed-models@r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
>

	[[alternative HTML version deleted]]


