[R] Small p-value good or bad?

[Ben Bolker]

T. Smithson <pillepop2003 <at> yahoo.de> writes:

> 
> Dear R-community,
> 
> I have a short question: How do I interpret the result of a
> likelihood ratio test correctly?
> 
> I am fitting a parametric survival model (with aftreg {eha}) and the
> output tells me the overall p-value of my model is < 0.001. My
> simple question is: Does the result mean my model fits the data well
> OR does it mean my model DOES NOT fit the data well?
> 
> Some side information how the p-value is calculated:
> 
> logtest <- -2 * (x$loglik[1] - x$loglik[2])
> pvalue <- 1-pchisq(logtest,df)
> 
> with
> x$loglik[1] = -274
> x$loglik[2] = -235
> df = 25
> 
> I know the answer would probably be "read the manual" but I found
> different opinions on the web and want to make sure I am
> interpreting it correctly.

  Well, the problem is the lack of context.  What TFM says about
'loglik' is:

 loglik: Vector of length two; first component is the value at the
          initial parameter values, the second componet is the
          maximized value.

The test being done here is asking whether the -2*L (deviance)
is significantly lower (corresponding to an improvement in model
fit) for the fitted parameters (maximized likelihood, or
minimized deviance) than for the initial parameter values.  The
problem is that I don't know without looking farther what
the initial parameters correspond to.  **If** aftreg starts
from a set of parameter values that correspond to a simpler/nested
model (e.g. all parameters for the effects of covariates set
to zero), *then* this p-value represents a p-value for the null
hypothesis that the fitted model is not a better fit to the data
than the simpler/nested model, in which case a small p-value
would allow you to reject the null hypothesis that the effects
of all covariates are equal to zero. In this case, the df should 
correspond to the increase in the number of parameters required
to fit the more complex model. However, this is just guessing
on the basis of a fairly standard procedure for this sort of test. If
this is indeed the context, then the procedure is not giving
you any information about whether the model fits the data well
(i.e., a goodness-of-fit test): it is telling you that the more complex
model fits the data *better* than the simpler model.

  If there are a variety of opinions on the web, by definition either
some of them are wrong *or* they are describing different contexts
(more likely).