[R] wilcox.test - difference between p-values of R and online calculators
peter dalgaard
pdalgd at gmail.com
Thu Sep 4 16:30:52 CEST 2014
Yes, that is the point that David made and that I illustrated with the simulations: The null distribution of W is more narrow in the presence of ties, hence W=485 is a more extreme observation in the tied case. I.e. it will look less extreme if you ignore that there are ties.
-pd
On 04 Sep 2014, at 15:17 , Lorenz, David <lorenz at usgs.gov> wrote:
> I think that the issue, at least with the online calculator that I looked
> at, is that it does not adjust the standard deviation of the test
> statistic for ties, so the standard deviation is larger and hence larger
> p-value. I was able to reproduce the reported z-score using the equation
> for the standard deviation with out ties.
> Dave
>
> Message: 14
>> Date: Wed, 3 Sep 2014 23:20:04 +0200
>> From: peter dalgaard <pdalgd at gmail.com
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=pdalgd@gmail.com>>
>> To: David L Carlson <dcarlson at tamu.edu
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=dcarlson@tamu.edu>>
>> Cc: "r-help at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=r-help@r-project.org>"
>> <r-help at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=r-help@r-project.org>>,
>> W Bradley Knox
>> <bradknox at mit.edu
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=bradknox@mit.edu>>
>> Subject: Re: [R] wilcox.test - difference between p-values of R and
>> online calculators
>> Message-ID: <FFDE9637-160E-4555-9C2A-E944947000D7 at gmail.com
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=FFDE9637-160E-4555-9C2A-E944947000D7@gmail.com>
>>>
>> Content-Type: text/plain; charset=us-ascii
>>
>> Notice that correct=TRUE for wilcox.test refers to the continuity
>> correction, not the correction for ties.
>>
>> You can fairly easily simulate from the exact distribution of W:
>>
>> x <- c(359,359,359,359,359,359,335,359,359,359,359,
>> 359,359,359,359,359,359,359,359,359,359,303,359,359,359)
>> y <- c(332,85,359,359,359,220,231,300,359,237,359,183,286,
>> 355,250,105,359,359,298,359,359,359,28.6,359,359,128)
>> R <- rank(c(x,y))
>> sim <- replicate(1e6,sum(sample(R,25))) - 325
>>
>> # With no ties, the ranks would be a permutation of 1:51, and we could do
>> sim2 <- replicate(1e6,sum(sample(1:51,25))) - 325
>>
>> In either case, the p-value is the probability that W >= 485 or W <= 165,
>> and
>>
>>> mean(sim >= 485 | sim <= 165)
>> [1] 0.000151
>>> mean(sim2 >= 485 | sim2 <= 165)
>> [1] 0.002182
>>
>> Also, try
>>
>> plot(density(sim))
>> lines(density(sim2))
>>
>> and notice that the distribution of sim is narrower than that of sim2
>> (hence the smaller p-value with tie correction), but also that the normal
>> approximationtion is not nearly as good as for the untied case. The
>> "clumpiness" is due to the fact that 35 of the ranks have the maximum value
>> of 34 (corresponding to the original 359's).
>>
>> -pd
>>
>> On 03 Sep 2014, at 19:13 , David L Carlson <dcarlson at tamu.edu
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=dcarlson@tamu.edu>>
>> wrote:
>>
>>> Since they all have the same W/U value, it seems likely that the
>> difference is how the different versions adjust the standard error for
>> ties. Here are a couple of posts addressing the issues of ties:
>>>
>>> http://tolstoy.newcastle.edu.au/R/e8/help/09/12/9200.html
>>>
>> http://stats.stackexchange.com/questions/6127/which-permutation-test-implementation-in-r-to-use-instead-of-t-tests-paired-and
>>>
>>> David C
>>>
>>> From: wbradleyknox at gmail.com
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=wbradleyknox@gmail.com>
>> [mailto:wbradleyknox at gmail.com
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=wbradleyknox@gmail.com>]
>> On Behalf Of W Bradley Knox
>>> Sent: Wednesday, September 3, 2014 9:20 AM
>>> To: David L Carlson
>>> Cc: Tal Galili; r-help at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=r-help@r-project.org>
>>> Subject: Re: [R] wilcox.test - difference between p-values of R and
>> online calculators
>>>
>>> Tal and David, thanks for your messages.
>>>
>>> I should have added that I tried all variations of true/false values for
>> the exact and correct parameters. Running with correct=FALSE makes only a
>> tiny change, resulting in W = 485, p-value = 0.0002481.
>>>
>>> At one point, I also thought that the discrepancy between R and these
>> online calculators might come from how ties are handled, but the fact that
>> R and two of the online calcultors reach the same U/W values seems to
>> indicate that ties aren't the issue, since (I believe) the U or W values
>> contain all of the information needed to calculate the p-value, assuming
>> the number of samples is also known for each condition. (However, it's been
>> a while since I looked into how MWU tests work, so maybe now's the time to
>> refresh.) If that's correct, the discrepancy seems to be based in what R
>> does with the W value that is identical to the U values of two of the
>> online calculators. (I'm also assuming that U and W have the same meaning,
>> which seems likely.)
>>>
>>> - Brad
>>>
>>> ____________________
>>> W. Bradley Knox, PhD
>>> http://bradknox.net<http://bradknox.net/>
>>> bradknox at mit.edu
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=bradknox@mit.edu>
>> <mailto:bradknox at mit.edu
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=bradknox@mit.edu>>
>>>
>>> On Wed, Sep 3, 2014 at 9:10 AM, David L Carlson <dcarlson at tamu.edu
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=dcarlson@tamu.edu>
>> <mailto:dcarlson at tamu.edu
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=dcarlson@tamu.edu>>>
>> wrote:
>>> That does not change the results. The problem is likely to be the way
>> ties are handled. The first sample has 25 values of which 23 are identical
>> (359). The second sample has 26 values of which 12 are identical (359). The
>> difference between the implementations may be a result of the way the ties
>> are ranked. For example the R function rank() offers 5 different ways of
>> handling the rank on tied observations. With so many ties, that could make
>> a substantial difference.
>>>
>>> Package coin has wilxon_test() which uses Monte Carlo simulation to
>> estimate the confidence limits.
>>>
>>> -------------------------------------
>>> David L Carlson
>>> Department of Anthropology
>>> Texas A&M University
>>> College Station, TX 77840-4352
>>>
>>>
>>> -----Original Message-----
>>> From: r-help-bounces at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=r-help-bounces@r-project.org>
>> <mailto:r-help-bounces at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=r-help-bounces@r-project.org>>
>> [mailto:r-help-bounces at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=r-help-bounces@r-project.org>
>> <mailto:r-help-bounces at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=r-help-bounces@r-project.org>>]
>> On Behalf Of Tal Galili
>>> Sent: Wednesday, September 3, 2014 5:24 AM
>>> To: W Bradley Knox
>>> Cc: r-help at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=r-help@r-project.org>
>> <mailto:r-help at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=r-help@r-project.org>>
>>> Subject: Re: [R] wilcox.test - difference between p-values of R and
>> online calculators
>>>
>>> It seems your numbers has ties. What happens if you run wilcox.test with
>>> correct=FALSE, will the results be the same as the online calculators?
>>>
>>>
>>>
>>> ----------------Contact
>>> Details:-------------------------------------------------------
>>> Contact me: Tal.Galili at gmail.com
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=Tal.Galili@gmail.com>
>> <mailto:Tal.Galili at gmail.com
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=Tal.Galili@gmail.com>>
>> |
>>> Read me: www.talgalili.com<http://www.talgalili.com> (Hebrew) |
>> www.biostatistics.co.il<http://www.biostatistics.co.il> (Hebrew) |
>>> www.r-statistics.com<http://www.r-statistics.com> (English)
>>>
>> ----------------------------------------------------------------------------------------------
>>>
>>>
>>>
>>> On Wed, Sep 3, 2014 at 3:54 AM, W Bradley Knox <bradknox at mit.edu
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=bradknox@mit.edu>
>> <mailto:bradknox at mit.edu
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=bradknox@mit.edu>>>
>> wrote:
>>>
>>>> Hi.
>>>>
>>>> I'm taking the long-overdue step of moving from using online
>> calculators to
>>>> compute results for Mann-Whitney U tests to a more streamlined system
>>>> involving R.
>>>>
>>>> However, I'm finding that R computes a different result than the 3
>> online
>>>> calculators that I've used before (all of which approximately agree).
>> These
>>>> calculators are here:
>>>>
>>>> http://elegans.som.vcu.edu/~leon/stats/utest.cgi
>>>> http://vassarstats.net/utest.html
>>>> http://www.socscistatistics.com/tests/mannwhitney/
>>>>
>>>> An example calculation is
>>>>
>>>>
>>>>
>> *wilcox.test(c(359,359,359,359,359,359,335,359,359,359,359,359,359,359,359,359,359,359,359,359,359,303,359,359,359),c(332,85,359,359,359,220,231,300,359,237,359,183,286,355,250,105,359,359,298,359,359,359,28.6,359,359,128))*
>>>>
>>>> which prints
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Wilcoxon rank sum test with continuity correction data: c(359, 359,
>> 359,
>>>> 359, 359, 359, 335, 359, 359, 359, 359, 359, and c(332, 85, 359, 359,
>> 359,
>>>> 220, 231, 300, 359, 237, 359, 183, 359, 359, 359, 359, 359, 359, 359,
>> 359,
>>>> 359, 303, 359, 359, and 286, 355, 250, 105, 359, 359, 298, 359, 359,
>> 359,
>>>> 28.6, 359, 359) and 359, 128) W = 485, p-value = 0.0002594 alternative
>>>> hypothesis: true location shift is not equal to 0 Warning message: In
>>>> wilcox.test.default(c(359, 359, 359, 359, 359, 359, 335, 359, : cannot
>>>> compute exact p-value with ties*
>>>>
>>>>
>>>> However, all of the online calculators find p-values close to 0.0025,
>> 10x
>>>> the value output by R. All results are for a two-tailed case.
>> Importantly,
>>>> the W value computed by R *does agree* with the U values output by the
>>>> first two online calculators listed above, yet it has a different
>> p-value.
>>>>
>>>> Can anyone shed some light on how and why R's calculation differs from
>> that
>>>> of these online calculators? Thanks for your time.
>>>>
>>>> ____________________
>>>> W. Bradley Knox, PhD
>>>> http://bradknox.net
>>>> bradknox at mit.edu
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=bradknox@mit.edu>
>> <mailto:bradknox at mit.edu
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=bradknox@mit.edu>>
>>>>
>>>> [[alternative HTML version deleted]]
>>>>
>>>> ______________________________________________
>>>> R-help at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=R-help@r-project.org>
>> <mailto:R-help at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=R-help@r-project.org>>
>> mailing list
>>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>>> PLEASE do read the posting guide
>>>> http://www.R-project.org/posting-guide.html
>>>> and provide commented, minimal, self-contained, reproducible code.
>>>>
>>>
>>> [[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> R-help at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=R-help@r-project.org>
>> <mailto:R-help at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=R-help@r-project.org>>
>> mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>>
>>>
>>> [[alternative HTML version deleted]]
>>>
>>> ______________________________________________
>>> R-help at r-project.org
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=R-help@r-project.org>
>> mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide
>> http://www.R-project.org/posting-guide.html
>>> and provide commented, minimal, self-contained, reproducible code.
>>
>> --
>> Peter Dalgaard, Professor,
>> Center for Statistics, Copenhagen Business School
>> Solbjerg Plads 3, 2000 Frederiksberg, Denmark
>> Phone: (+45)38153501
>> Email: pd.mes at cbs.dk
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=pd.mes@cbs.dk> Priv:
>> PDalgd at gmail.com
>> <https://mail.google.com/mail/?view=cm&fs=1&tf=1&to=PDalgd@gmail.com>
>>
>>
>
> [[alternative HTML version deleted]]
>
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
--
Peter Dalgaard, Professor,
Center for Statistics, Copenhagen Business School
Solbjerg Plads 3, 2000 Frederiksberg, Denmark
Phone: (+45)38153501
Email: pd.mes at cbs.dk Priv: PDalgd at gmail.com
More information about the R-help
mailing list