[R-sig-eco] How to obtain P value from Monte Carlo sampling for adonis (permanova)?

[Cade, Brian]

Ellen:  If you are running permutation procedures with data that have very
small sample sizes in each group (your two groups of n = 3 each yields only
6!/(3!3!) = M = 20 permutations under Ho), then you just have to live with
the fact that the smallest possible P-value is 1/M (= 0.05 for your two
group example).  There is nothing magical about P < 0.05 anyways.  But as
the Monte Carlo resampling approach to obtaining permutation P-values
really is just a method to attempt to approximate the exact permutation
P-value (and usually used when M is so large that you can not enumerate it
exactly in reasonable computation time), you should not rely on it when the
number or permutations M is so small, and especially not just because you
might obtain a P < 1/M.  If you obtain a P-value <0.05 in your example
using the Monte Carlo resampling procedure, all that indicates is that the
Monte Carlo resampling approach is a poor approximation in this small
sample situation.  I think it is always preferable to obtain and interpret
the exact permutation distribution if it is easily calculable.  Using a
crummy approximation just because you want P < 0.05 seems unreasonable to
me.

Brian

Brian S. Cade, PhD

U. S. Geological Survey
Fort Collins Science Center
2150 Centre Ave., Bldg. C
Fort Collins, CO  80526-8818

email:  cadeb at usgs.gov <brian_cade at usgs.gov>
tel:  970 226-9326


On Fri, Nov 18, 2016 at 1:39 AM, Ellen Pape <ellen.pape at gmail.com> wrote:

> Dear all,
>
> I have recently decided to switch from Permanova/Primer to R, because the
> latter is freeware (and I don't know for how long I will still have a
> license). However, if I cannot seem to resolve my problem (see below), I
> might have to go back to using Primer/Permanova.
>
> If I run pairwise permanova/adonis tests on my data, the number of unique
> permutations is small (I have two groups, each group has 3 observations)
> and the minimum P value I can get is larger than the alpha value I (and
> most people) that I use to determine statistical significance (i.e. 0.05).
> In the manual of the PERMANOVA+ add-on in Primer by Anderson et al. (2008)
>  it is mentioned (page 28 and onwards) that when the number of unique
> permutations is small (<100) than one should preferably interpret the
> Monte-Carlo p value.
>
> Does anyone know how to do this in R?
>
>
> In my internet search I stumbled upon this page:
> http://r.789695.n4.nabble.com/monte-carlo-simulations-in-
> permanova-in-vegan-package-td4714034.html.
> "JAri Oksanen answered here: 2. If you want to do so, you can generate your
> resampling matrices by hand and use that matrix as the argument of
> permutations=. See the documentations (?adonis) which tells how to do so.
> ", but I don't understand how to this..
>
> Thank you very much!
>
> Ellen
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-ecology mailing list
> R-sig-ecology at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
>
>

	[[alternative HTML version deleted]]