[R] Multiple comparisons in a non parametric case

[Rolf Turner]

It looks to me like what you are doing is trying to judge
significance of differences by non-overlap of single-sample
confidence intervals.  While this is appealing, it's not quite
right.

I just looked into my copy of Applied Nonparametric Statistics
(second ed.) by Wayne W. Daniel (Duxbury, 1990) but that
only deals with the situation where there is a single replicate
per block-treatment combination (whereas you have 10 reps)
and block-treatment interaction is assumed to be non-existent.

The method that Daniel prescribes in this simple setting seems to be
no more than applying the Bonferroni method of multiple comparisons.
(Daniel does not say; his book is very much a cook-book.)  So you
might simply try Bonferroni --- i.e. do all k-choose-2 pairwise
comparisons between treatments (using the appropriate 2 sample method
for each comparison) doing each comparison at the alpha/k-choose-2
significance level.  Where k = the number of treatments = 4 in your
case.  This method is not going to be super-powerful but it is
sometimes surprizing how well Bonferroni stacks up against more
``sophisticated'' methods.

Daniel gives a reference to ``Nonparametric Statistical Methods'' by
Myles Hollander and Douglas A. Wolfe, New York, Wiley, 1973, for ``an
alternative multiple comparisons formula''.  I don't have this book,
and don't know what direction Hollander and Wolfe ride off in, but it
***might*** be worth trying to get your hands on it and see.

Finally --- in what way are the assumptions of Anova violated?  The
conventional wisdom is that Anova is actually quite robust to
non-normality.  Particularly when the sample size is large --- and 10
reps per treatment combination is pretty good.  Heteroskedasticity is
more of a worry, but it's not so much of a worry when the design is
nicely balanced.  As yours is.  And finally-finally --- have you
tried transforming your data to make them a bit more normal and/or
homoskedastic?

I hope this is some help.

				cheers,

					Rolf Turner
					rolf at math.unb.ca

Marco Chiarandini wrote:

> I am conducting a full factorial analysis. I have one factor
> consisting in algorithms, which I consider my treatments, and another
> factor made of the problems I want to solve. For each problem I
> obtain a response variable which is stochastic. I replicate the
> measure of this response value 10 times.
> 
> When I apply ANOVA the assumptions do not hold, hence I must rely on
> non parametric tests.
> 
> By transforming the response data in ranks, the Friedman test tells
> me that there is statistical significance in the difference of the
> sum of ranks of at least one of the treatments.
> 
> I would like now to produce a plot for the multiple comparisons
> similar to the Least Significant Difference or the Tukey's Honest
> Significant Difference used in ANOVA. Since I am in the non
> parametric case I can not use these methods.
> 
> Instead, I compare graphically individual treatments by plotting the
> sum of ranks of each treatment togehter with the 95% confidence
> interval. To compute the interval I use the Friedman test as
> suggested by Conover in "Practical Nonparametric statistics".
> 
> I obtain something like this:
> 
> Treat. A                |-+-|
> Treat. B              |-+-|
> Treat. C                   |-+-|
> Treat. D           |-+-|
> 
> The intervals have all the same spread because the number of
> replications was the same for all experimental units.
> 
> I would like to know if someone in the list had a similar experience
> and if what I am doing is correct. In alternative also a reference to
> another list which could better fit my request is welcome.