[R] Calibration score for survival probability

[Eleni Rapsomaniki]

Dear Prof. Harrell,
Thank you very much for your prompt and very helpful response. I guess
that since a global statistic such as a chi-sq test is not applicable in
this case, the calibration curve itself from the calibration() is the
most informative alternative (most graphical methods reveal more
information than a single statistic anyway!). I will try the updated
version in the rms package to compare.

Best Wishes 

Eleni Rapsomaniki
 
Research Associate
Strangeways Research Laboratory
Department of Public Health and Primary Care
University of Cambridge
 

-----Original Message-----
From: Frank E Harrell Jr [mailto:f.harrell at vanderbilt.edu] 
Sent: 23 November 2009 13:01
To: Eleni Rapsomaniki
Cc: r-help at r-project.org
Subject: Re: [R] Calibration score for survival probability

Eleni Rapsomaniki wrote:
> Good afternoon!
> 
> I need to evaluate the goodness-of-fit (aka calibration) for survival
probability estimates from a Cox model. 
> I tried to use 'calibrate' in the Design package but I'm not sure if
it should/would produce what I need (ie a chi-sq type statistic with a
table of expected vs observed probabilities). Any other functions I
should be aware of?
> 
> Also, has anybody come across an implementation of the statistic
described in:
> "A global goodness of fit statistic for Cox regression models" by
Parzen & Lpisitz, Biometrics 55, 1999 
> 
> Many thanks in advance
> 
> Eleni Rapsomaniki

Eleni,

The Design package, and its replacement, the rms package, produces 
calibration curves but no chi-square test because we do not have a 
corresponding method for that.  Formal tests are overused in this 
context anyway.  An index such as the maximum or 90th percentile of 
absolute calibration error are often more useful.  I have learned 
however that any statistical method that categorizes continuous 
variables (in this case, the predictions or the covariate space) is 
arbitrary and has many other problems.  The calibrate functions in the 
rms package have a new option to obtain smooth calibration curves 
without grouping, by fitting spline hazard models during validation.

Note that if you have done any model/variable selection you have to 
re-run such model building from scratch for each resample of the data, 
or the calibration plot will be over optimistic.  calibrate() makes this

automatic if doing backward stepdown variable selection.  Many 
statisticians make the mistake of only "validating" the final selected 
model, which can only be done by one-time data splitting (which requires

  tens of thousands of observations to perform adequately).

Frank

> 
> Research Associate
> Strangeways Research Laboratory
> Department of Public Health and Primary Care
> 
> University of Cambridge
>  
> 
> ______________________________________________
> R-help at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
> 


-- 
Frank E Harrell Jr   Professor and Chair           School of Medicine
                      Department of Biostatistics   Vanderbilt
University