[R] What is a "good fit" Brier score and Harrel's C Index

[E Joffe]

Thank you so much for this answer.
Per your comment I have moved the discussion (including your answer) to
StackExchange
At
http://stats.stackexchange.com/questions/71417/what-is-a-good-fit-brier-scor
e-and-harrels-c-index 

Many thanks !
Erel

-----Original Message-----
From: David Winsemius [mailto:dwinsemius at comcast.net] 
Sent: Saturday, September 28, 2013 5:04 PM
To: E Joffe
Cc: r-help at r-project.org
Subject: Re: [R] What is a "good fit" Brier score and Harrel's C Index


On Sep 28, 2013, at 8:14 AM, E Joffe wrote:

>
> Hi all,
>
> I am evaluating survival models using Brier score ("peperr") and 
> Harrel's C-Index ("Hmisc").

It's spelled 'Harrell'.

> I am wondering:
>
> 1. What would be considered a "good fit" according to these scores 
> (like the heuristic levels we have for R square in linear regressions) 
> ?
>
> 2. Are there any papers to cite on the matter (I couldn't find any) ?

Frank Harrell's excellent text "Regression Modeling Strategies" has an
extensive discussion of "goodness of fit" and the principles of model
comparison. It's both too involved as well as off-topic for Rhelp. The other
text to consult is Steyerberg's "Clinical Prediction Models".

>
> 3. Is there any paper to cite that discusses the limitation of using 
> traditional reporting for model fit in survival analysis as opposed to 
> these measures  ?

I predict that the RMS bibliography would be an excellent place to start
your search.

Despite getting his name attached to what he calls the 'c-index', I don't
think one could call Frank Harrell a proponent of that measure or any of the
"competitors". It's really just a dressed up/transformed AUC. The message I
have taken from reading his book and listening to presentations is that one
should apply biologic tests of sensibility as well as careful investigation
of the functional relationships between candidate predictors and the
outcomes of interest. He speaks very disparagingly about automatic
procedures.

-- 

David Winsemius, MD
Alameda, CA, USA