[R-sig-eco] How to obtain P value from Monte Carlo sampling for adonis (permanova)?

[Law, Jason]

Brian has some really sound advice here. Using a Monte Carlo approximation rather than the exact result kind of misses the entire point. 0.05 is arbitrary and using an approximation to an exact distribution that you can easily calculate is misguided. Getting a p-value of 0.1 versus 0.05 really shouldn't result in any groundbreaking scientific differences in interpretation, despite what a journal might say about your p-values.

You should read Greenland et al (http://link.springer.com/article/10.1007/s10654-016-0149-3) which has some choice quotes from Neyman and Pearson and a lot of wisdom regarding the insanity surrounding our use of p-values.

Jason Law
Statistician, City of Portland
Water Pollution Control Laboratory
6543 N Burlington Ave,
Portland, OR

503-823-1038
jason.law at portlandoregon.gov



-----Original Message-----
From: R-sig-ecology [mailto:r-sig-ecology-bounces at r-project.org] On Behalf Of Cade, Brian
Sent: Friday, November 18, 2016 8:51 AM
To: Ellen Pape <ellen.pape at gmail.com>
Cc: <r-sig-ecology at r-project.org> <r-sig-ecology at r-project.org>
Subject: Re: [R-sig-eco] How to obtain P value from Monte Carlo sampling for adonis (permanova)?

Ellen:  If you are running permutation procedures with data that have very small sample sizes in each group (your two groups of n = 3 each yields only
6!/(3!3!) = M = 20 permutations under Ho), then you just have to live with the fact that the smallest possible P-value is 1/M (= 0.05 for your two group example).  There is nothing magical about P < 0.05 anyways.  But as the Monte Carlo resampling approach to obtaining permutation P-values really is just a method to attempt to approximate the exact permutation P-value (and usually used when M is so large that you can not enumerate it exactly in reasonable computation time), you should not rely on it when the number or permutations M is so small, and especially not just because you might obtain a P < 1/M.  If you obtain a P-value <0.05 in your example using the Monte Carlo resampling procedure, all that indicates is that the Monte Carlo resampling approach is a poor approximation in this small sample situation.  I think it is always preferable to obtain and interpret the exact permutation distribution if it is easily calculable.  Using a crummy approximation just because you want P < 0.05 seems unreasonable to me.

Brian

Brian S. Cade, PhD

U. S. Geological Survey
Fort Collins Science Center
2150 Centre Ave., Bldg. C
Fort Collins, CO  80526-8818

email:  cadeb at usgs.gov <brian_cade at usgs.gov>
tel:  970 226-9326


On Fri, Nov 18, 2016 at 1:39 AM, Ellen Pape <ellen.pape at gmail.com> wrote:

> Dear all,
>
> I have recently decided to switch from Permanova/Primer to R, because 
> the latter is freeware (and I don't know for how long I will still 
> have a license). However, if I cannot seem to resolve my problem (see 
> below), I might have to go back to using Primer/Permanova.
>
> If I run pairwise permanova/adonis tests on my data, the number of 
> unique permutations is small (I have two groups, each group has 3 
> observations) and the minimum P value I can get is larger than the 
> alpha value I (and most people) that I use to determine statistical significance (i.e. 0.05).
> In the manual of the PERMANOVA+ add-on in Primer by Anderson et al. 
> (2008)  it is mentioned (page 28 and onwards) that when the number of 
> unique permutations is small (<100) than one should preferably 
> interpret the Monte-Carlo p value.
>
> Does anyone know how to do this in R?
>
>
> In my internet search I stumbled upon this page:
> http://r.789695.n4.nabble.com/monte-carlo-simulations-in-
> permanova-in-vegan-package-td4714034.html.
> "JAri Oksanen answered here: 2. If you want to do so, you can generate 
> your resampling matrices by hand and use that matrix as the argument 
> of permutations=. See the documentations (?adonis) which tells how to do so.
> ", but I don't understand how to this..
>
> Thank you very much!
>
> Ellen
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-ecology mailing list
> R-sig-ecology at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
>
>

	[[alternative HTML version deleted]]

_______________________________________________
R-sig-ecology mailing list
R-sig-ecology at r-project.org
https://stat.ethz.ch/mailman/listinfo/r-sig-ecology