[R] p-values

Thu Aug 11 05:01:48 CEST 2005

	  pperm seems reasonable, though I have not looked at the details.

	  We should be careful about terminology, however.  So-called "exact 
p-values" are generally p-values computed assuming a distribution over a 
finite set of possible outcomes assuming some constraints to make the 
outcome space finite.  For example, Fisher's exact test for a 2x2 table 
assumes the marginals are fixed.

	  I don't remember the details now, but I believe there is literature 
claiming that this may not be the best thing to do when, for example, 
when it is reasonable to assume that the number in each cell is Poisson. 
  In such cases, you may lose statistical power by conditioning on the 
marginals.  I hope someone else will enlighten us both, because I'm not 
current on the literature in this area.

	  The situation with "exact tests" and "exact p-values" is not nearly 
as bad as with so-called ""exact confidence intervals", which promise to 
deliver at least the indicated coverage probability.  With discrete 
distributions, it is known that 'Approximate is better than "exact' for 
interval estimation of binomial proportions', as noted in an article of 
this title by A. Agresti and B. A. Coull (1998) American Statistician, 
52:  119-126.  (For more on this particular issue, see Brown, Cai and 
Dasgupta 2003 "Interval Estimation in Exponential Families", Statistica 
Sinica 13:  19-49).

	  If this does not answer your question adequately, may I suggest you 
try the posting guide.  People report having found answers to difficult 
questions in the process of preparing a question following that guide, 
and when they do post a question, they are much more likely to get a 
useful reply.

	  spencer graves

Peter Ho wrote:

> Spencer,
> 
> 
> Thank you for referring me to your other email on Exact goodness-of-fit 
> test. However, I'm not entirely sure if what you mentioned is the same 
> for my case. I'm not a statistician and it would help me if you could 
> explain what you meant in a little more detail. Perhaps I need to 
> explain the problem in more detail.
> 
> I am looking for a way to calculate exaxt p-values by Monte Carlo 
> Simulation for Durbin's test. Durbin's test statistic is similar to 
> Friedman's statistic, but considers the case of Balanced Incomplete 
> block designs. I have found a function written by Felipe de Mendiburu 
> for calculating Durbin's statistic, which gives the chi-squared p-value. 
> I have also been read an article by Torsten Hothorn "On exact rank Tests 
> in R" (R News 1(1), 11–12.) and he has shown how to calculate Monte 
> Carlo p-values using pperm. In the article by Torsten Hothorn he gives:
> 
> R> pperm(W, ranks, length(x))
> 
> He compares his method to that of StatXact, which is the program Rayner 
> and Best suggested using. Is there a way to do this for example for the 
> friedman test.
> 
> A paper by Joachim Rohmel discusses "The permutation distribution for 
> the friendman test" (Computational Statistics & Data Analysis 1997, 26: 
> 83-99). This seems to be on the lines of what I need, although I am not 
> quite sure. Has anyone tried to recode his APL program for R?
> 
> I have tried a number of things, all unsucessful. Searching through 
> previous postings have not been very successful either. It seems that 
> pperm is the way to go, but I would need help from someone on this.
> 
> Any hints on how to continue would be much appreciated.
> 
> 
> Peter
> 
> 
> Spencer Graves wrote:
> 
>> Hi, Peter:
>>
>>       Please see my reply of a few minutes ago subject:  exact 
>> goodness-of-fit test.  I don't know Rayner and Best, but the same 
>> method, I think, should apply.  spencer graves
>>
>> Peter Ho wrote:
>>
>>  
>>
>>> HI R-users,
>>>
>>> I am trying to repeat an example from Rayner and Best "A contingency 
>>> table approach to nonparametric testing (Chapter 7, Ice cream example).
>>>
>>> In their book they calculate Durbin's statistic, D1, a dispersion 
>>> statistics, D2, and a residual. P-values for each statistic is 
>>> calculated from a chi-square distribution and also Monte Carlo p-values.
>>>
>>> I have found similar p-values based on the chi-square distribution by 
>>> using:
>>>
>>> > pchisq(12, df= 6, lower.tail=F)
>>> [1] 0.0619688
>>> > pchisq(5.1, df= 6, lower.tail=F)
>>> [1] 0.5310529
>>>
>>> Is there a way to calculate the equivalent Monte Carlo p-values?
>>>
>>> The values were 0.02 and 0.138 respectively.
>>>
>>> The use of the approximate chi-square probabilities for Durbin's test 
>>> are considered not good enough according to Van der Laan (The 
>>> American Statistician 1988,42,165-166).
>>>
>>>
>>> Peter
>>> --------------------------------
>>> ESTG-IPVC
>>>
>>> ______________________________________________
>>> R-help at stat.math.ethz.ch mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-help
>>> PLEASE do read the posting guide! 
>>> http://www.R-project.org/posting-guide.html
>>>   
>>
>>
>>  
>>
> 

-- 
Spencer Graves, PhD
Senior Development Engineer
PDF Solutions, Inc.
333 West San Carlos Street Suite 700
San Jose, CA 95110, USA

spencer.graves at pdf.com
www.pdf.com <http://www.pdf.com>
Tel:  408-938-4420
Fax: 408-280-7915