[R] On Corrections for Chi-Sq Goodness of Fit Test

[Michael Fuller]

Hi Rolf,
Thank you for clarifying. After reading the help page over more carefully, I realize that I misunderstood what chisq.test. was doing. I thought it was using simulation whenever the expected frequencies were given using the "p" argument, whereas it used the asymptotic chi-squared distribution of the test statistic when "p"was not given. I now see that, based on the help page, the asymptotic distribution is always used, unless simulate.p.value is TRUE.

Mike

On Dec 25, 2011, at 3:06 AM, Rolf Turner wrote:

> 
> When the p-value is calculated via simulation the test is not, strictly
> speaking a ``chi-squared test''.    That is, it is *not* assumed that the
> distribution of the test statistic, under the null hypothesis, is a chi-squared
> distribution.  Instead an empirical distribution derived from simulating
> samples under the null hypothesis is used.  The *same statistic* is calculated,
> but the chi-squared distribution is never invoked.  The simulated p-value
> facility is provided so that tests can be conducted even when the requirements
> for the validity of the chi-squared (null) distribution are not met.
> 
> The test, whether you want to call it a chi-squared test or not, is a perfectly
> valid goodness of fit test.  (Modulo the caveat that I previously mentioned,
> i.e. that it is required that there be no ties amongst the values of the test
> statistics.  Which is a mild worry, but has nothing to do with continuity corrections
> which are meaningless in this context.)
> 
>    cheers,
> 
>        Rolf Turner
> 
> On 25/12/11 08:26, Michael Fuller wrote:
>> On Dec 22, 2011, at 8:56 PM, Rolf Turner wrote:
>> 
>>> On 20/12/11 10:24, Michael Fuller wrote:
>>>> TOPIC
>>>> My question regards the philosophy behind how R implements corrections to chi-square statistical tests. At least in recent versions (I'm using 2.13.1 (2011-07-08) on OSX 10.6.8.), the chisq.test function applies the Yates continuity correction for 2 by 2 contingency tables. But when used as a goodness of fit test (GoF, aka likelihood ratio test), chisq.test does not appear to implement any corrections for widely recognized problems, such as small sample size, non-uniform expected frequencies, and one D.F.
>>>> 
>>>>> From the help page:
>>>> "In the goodness-of-fit case simulation is done by random sampling from the discrete distribution specified by p, each sample being of size n = sum(x)."
>>>> 
>>>> Is the thinking that random sampling completely obviates the need for corrections?
>>>    Yes.
>>>> Wouldn't the same statistical issues still apply
>>>    No.
>>>> (e.g. poor continuity approximation with one D.F.,
>>>    There are no degrees of freedom involved.  There is no continuity involved.
>>>    The observed test statistics (say "Stat") is compared with a number of
>>>    test statistics, Stat_1, ..., Stat_N, calculated from data sets simulated under
>>>    the null hypothesis.  If the null is true, then Stat and Stat_1, ...., Stat_N are
>>>    all of ``equal status''.  If there are m values of the Stat_i which are greater
>>>    than Stat, then the ``probability of observing, under the null hypothesis,
>>>    data as extreme as, or more extreme than, what you actually observed''
>>>    is the probability of randomly selecting one of a specified set of m+1 ``slots''
>>>    out of a total of N+1 slots (where each slot has probability 1/(N+1)).
>> But if the test is truly a chi-square Goodness of Fit (GoF) test, then:
>> 
>> (1) the test compares the Stat to a chi-square distribution with (k-1) degrees of
>> freedom, for k frequency categories.
>> 
>> (2) the shape of the distribution depends on the degrees of freedom
>> 
>> (3) continuity is an issue, because the values for Stat are discrete, whereas
>> the chi-square distribution is continuous. Therefore the p-value is not exact.
>> 
>> If I understand your description, the function generates a probability
>> distribution by sampling a set of values with equal probability. The probability of
>> any value in the final distribution depends on its frequency in the sample set. This
>> is a standard Monte Carlo method, but the PDF it generates is not chi-square (the
>> resulting distribution depends upon the distribution of the sample set). Clearly,
>> using a permutation method such as this to determine the statistical significance of
>> a given test statistic is non-parametric and in particular, not chi-square.
>> 
>> If what you say is true, then the chisq.test function in R does not implement a true GoF
>> test. I don't understand why the GoF test is not computed the standard way. It should
>> compute the chi-square statistic, using the values of p to generate expected
>> frequencies. And it should use a continuity correction when degrees of freedom = 1.
>> 
>> Thank you for taking the time to respond to my message.
>> 
>> Cheers,
>> Mike
>> 
>>>    Thus the p-value is (exactly) equal to (m+1)/(N+1).
>>> 
>>>    The only restriction is that there be no ties amongst the values of Stat
>>>    and Stat_1, ..., Stat_N.  There being ties is of fairly low probability, but is
>>>    not of zero probability --- since there is a finite number of possible samples
>>>    and hence of statistic values.  So this restriction is a mild worry.
>>> 
>>>    However a ``continuity correction'' would be of no help whatsoever.
>>>> problems with non-uniform expected frequencies, etc) with random sampling?
>>>    Don't understand what you mean by this.
>>> 
>>>        cheers,
>>> 
>>>            Rolf Turner
>> ====================
>> Michael M. Fuller, Ph.D.
>> Department of Biology
>> University of New Mexico
>> Albuquerque, NM
>> EMAIL: mmfuller at unm.edu
>> WEB: biology.unm.edu/mmfuller
>> 
>> 
>> 
>> 
>> 
> 

====================
Michael M. Fuller, Ph.D.
Department of Biology
University of New Mexico
Albuquerque, NM
EMAIL: mmfuller at unm.edu
WEB: biology.unm.edu/mmfuller