[R-sig-eco] posthoc tests beyond Tukey's HSD

[Simon Blomberg]

On Mon, 2008-07-14 at 22:20 -0700, Jarrett Byrnes wrote:
> I'm currently in a bit of a quandry.  While I feel I've got a good  
> feel for using the multcomp package for posthoc tests, I've been left  
> with two questions - mainly by colleagues who read my results (and, I  
> fear, reviewers in the future, as the results from glht are not  
> something typically repoprted in many journals).
> 
> To whit, I have two questions
> 
> 1) How do _you_ report results from the multcomp package?  Do you have  
> a favorite adjustments to use (I've become rather fond of the False  
> Discovery Rate after reading several recent papers about it, as well  
> as it's cousin, the sharpened False Discovery rate, for which I've  
> whipped up some code if anyone would like it)?   What would you find  
> acceptable and understandable as a reviewer?

Report what test you used, and your justification for using it. Also try
to provide confidence intervals for effects, not just hypothesis tests.
They are of more use in the long run.

Multiple comparisons procedures have to be some of the most abused
statistical methods. I avoid them wherever possible. The problem is that
scientists often carry out a well-designed experiment to test some
well-defined a priori hypotheses. Then they ignore the logical structure
of the experiment and use an MCP to conduct tests that should have been
defined a priori, and usually also a set of other comparisons that are
not even interesting. The Neyman-Pearson theory works perfectly fine for
a small set of a priori hypotheses. You don't have to correct for
multiple comparisons in this case, and you will get more power if you
don't. I wish scientists would have the courage of their convictions and
defend (and only test) their a priori hypotheses. John Nelder has some
provocative views on this and other subjects (with which I agree!) See:
The Statistician (1999) 48(2): 257-269.

Having said that, there is a limited use for MCPs, in the context of
looking at the data and then deciding which tests might be interesting
(ie fishing expeditions). The use of FDR methods also makes sense for
example with microarray data where there are thousands of tests and no
clearly defined hypotheses, other than "we think there should be
differences for some subset of genes, somewhere."

> 
> 2) But, sometimes, you want your old Dunnett's test and Ryan's Q - or,  
> really, any of the tests from the wonderful Day and Quinn 1989  
> Ecological Monographs paper.  I've not had much luck looking beyond  
> Tukey's HSD.  Has anyone found any good code for some of these other  
> common posthocs?  I admit, I might just be missing some crucial  
> package, although I've looked around a good bit.
> 

There are two issues: 1. What kind of contrasts do you want to make? and
2. What kind of adjustment do you want to do? You can get a variety of
contrasts by using the type= argument to mcp(), which then passes it
along to contrMat (see ?contrMat). You can get different sorts of
adjustment by using the test=adjusted() argument to summary.glht.
See ?adjusted. adjusted also accepts as arguments the values in
p.adjust.methods. See ?p.adjust.methods.

The documentation for the multcomp package is really very good. There
are a couple of pdf documents in the multcomp directory that are very
much worth reading.

Cheers,

Simon.

> 
> Thanks, and I look forward to your answers!
> 
> -Jarrett
> 
> 
> 
> 
> ----------------------------------------
> Jarrett Byrnes
> Postdoctoral Associate, Santa Barbara Coastal LTER
> Marine Science Institute
> University of California Santa Barbara
> Santa Barbara, CA 93106-6150
> 
> 
> 	[[alternative HTML version deleted]]
> 
> _______________________________________________
> R-sig-ecology mailing list
> R-sig-ecology at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
-- 
Simon Blomberg, BSc (Hons), PhD, MAppStat. 
Lecturer and Consultant Statistician 
Faculty of Biological and Chemical Sciences 
The University of Queensland 
St. Lucia Queensland 4072 
Australia
Room 320 Goddard Building (8)
T: +61 7 3365 2506
http://www.uq.edu.au/~uqsblomb
email: S.Blomberg1_at_uq.edu.au

Policies:
1.  I will NOT analyse your data for you.
2.  Your deadline is your problem.

The combination of some data and an aching desire for 
an answer does not ensure that a reasonable answer can 
be extracted from a given body of data. - John Tukey.