[R-sig-eco] How to obtain P value from Monte Carlo sampling for adonis (permanova)?

[Ellen Pape]

Ok! Thank you all for your advice! This really helps! I will not be using
Monte Carlo approx to obtain the P value for the data that I have..

On 18 November 2016 at 20:39, Law, Jason <Jason.Law at portlandoregon.gov>
wrote:

> Brian has some really sound advice here. Using a Monte Carlo approximation
> rather than the exact result kind of misses the entire point. 0.05 is
> arbitrary and using an approximation to an exact distribution that you can
> easily calculate is misguided. Getting a p-value of 0.1 versus 0.05 really
> shouldn't result in any groundbreaking scientific differences in
> interpretation, despite what a journal might say about your p-values.
>
> You should read Greenland et al (http://link.springer.com/
> article/10.1007/s10654-016-0149-3) which has some choice quotes from
> Neyman and Pearson and a lot of wisdom regarding the insanity surrounding
> our use of p-values.
>
> Jason Law
> Statistician, City of Portland
> Water Pollution Control Laboratory
> 6543 N Burlington Ave,
> Portland, OR
>
> 503-823-1038
> jason.law at portlandoregon.gov
>
>
>
> -----Original Message-----
> From: R-sig-ecology [mailto:r-sig-ecology-bounces at r-project.org] On
> Behalf Of Cade, Brian
> Sent: Friday, November 18, 2016 8:51 AM
> To: Ellen Pape <ellen.pape at gmail.com>
> Cc: <r-sig-ecology at r-project.org> <r-sig-ecology at r-project.org>
> Subject: Re: [R-sig-eco] How to obtain P value from Monte Carlo sampling
> for adonis (permanova)?
>
> Ellen:  If you are running permutation procedures with data that have very
> small sample sizes in each group (your two groups of n = 3 each yields only
> 6!/(3!3!) = M = 20 permutations under Ho), then you just have to live with
> the fact that the smallest possible P-value is 1/M (= 0.05 for your two
> group example).  There is nothing magical about P < 0.05 anyways.  But as
> the Monte Carlo resampling approach to obtaining permutation P-values
> really is just a method to attempt to approximate the exact permutation
> P-value (and usually used when M is so large that you can not enumerate it
> exactly in reasonable computation time), you should not rely on it when the
> number or permutations M is so small, and especially not just because you
> might obtain a P < 1/M.  If you obtain a P-value <0.05 in your example
> using the Monte Carlo resampling procedure, all that indicates is that the
> Monte Carlo resampling approach is a poor approximation in this small
> sample situation.  I think it is always preferable to obtain and interpret
> the exact permutation distribution if it is easily calculable.  Using a
> crummy approximation just because you want P < 0.05 seems unreasonable to
> me.
>
> Brian
>
> Brian S. Cade, PhD
>
> U. S. Geological Survey
> Fort Collins Science Center
> 2150 Centre Ave., Bldg. C
> Fort Collins, CO  80526-8818
>
> email:  cadeb at usgs.gov <brian_cade at usgs.gov>
> tel:  970 226-9326
>
>
> On Fri, Nov 18, 2016 at 1:39 AM, Ellen Pape <ellen.pape at gmail.com> wrote:
>
> > Dear all,
> >
> > I have recently decided to switch from Permanova/Primer to R, because
> > the latter is freeware (and I don't know for how long I will still
> > have a license). However, if I cannot seem to resolve my problem (see
> > below), I might have to go back to using Primer/Permanova.
> >
> > If I run pairwise permanova/adonis tests on my data, the number of
> > unique permutations is small (I have two groups, each group has 3
> > observations) and the minimum P value I can get is larger than the
> > alpha value I (and most people) that I use to determine statistical
> significance (i.e. 0.05).
> > In the manual of the PERMANOVA+ add-on in Primer by Anderson et al.
> > (2008)  it is mentioned (page 28 and onwards) that when the number of
> > unique permutations is small (<100) than one should preferably
> > interpret the Monte-Carlo p value.
> >
> > Does anyone know how to do this in R?
> >
> >
> > In my internet search I stumbled upon this page:
> > http://r.789695.n4.nabble.com/monte-carlo-simulations-in-
> > permanova-in-vegan-package-td4714034.html.
> > "JAri Oksanen answered here: 2. If you want to do so, you can generate
> > your resampling matrices by hand and use that matrix as the argument
> > of permutations=. See the documentations (?adonis) which tells how to do
> so.
> > ", but I don't understand how to this..
> >
> > Thank you very much!
> >
> > Ellen
> >
> >         [[alternative HTML version deleted]]
> >
> > _______________________________________________
> > R-sig-ecology mailing list
> > R-sig-ecology at r-project.org
> > https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
> >
> >
>
>         [[alternative HTML version deleted]]
>
> _______________________________________________
> R-sig-ecology mailing list
> R-sig-ecology at r-project.org
> https://stat.ethz.ch/mailman/listinfo/r-sig-ecology
>

	[[alternative HTML version deleted]]