[R] net classification improvement?

[Vincent Vinh-Hung]

Greetings,
Regarding "NRI requires cutoff values" or not, the more recent Pencina
et al's paper suggests the use of a category-free NRI, i.e. without
cutoffs.
Pencina, D'Agostino, Steyerberg. Statist Med  2011,30:11-21.
For computations, I find Hmisc's improveProb is very flexible. Allows
use for category-based or category-free NRI, and also gives IDI
results.
Vincent

> Message: 48
> Date: Tue, 17 Jan 2012 16:55:51 +0000
> From: "Essers, Jonah" <Jonah.Essers at childrens.harvard.edu>
> To: "Kevin E. Thorpe" <kevin.thorpe at utoronto.ca>
> Cc: "r-help at R-project.org" <r-help at r-project.org>
> Subject: Re: [R] net classification improvement?
> Message-ID: <CB3B10E7.B41C%jonah.essers at childrens.harvard.edu>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Actually, I don't think I made myself clear and I wrote this late last
> night....Sorry. More the issue is that the raw model predictions (from 0
> to 1) have no inherent clinical value to them. I.e. They aren't "risk of
> disease" or "risk of outcome". They are raw scores that are specific to
> each model and are meant to discriminate one disease from another disease.
> Trying to compare models is impossible because the NRI requires cutoff
> values.  The cutoffs are different for each model.
>
> So, as I've done more reading, it appears the the IRI--Integrated
> Discrimination Improvement Index--which is na?ve to cutoff values--may be
> more what I'm looking for. Does this make sense? I guess I just need a
> sanity check.
>
> I have been toying with the PredictABEL package and this seems to like my
> data inputs just fine and relies on HMISC and ROCR, both packages I know
> well.
>
> Thanks
> jonah
>
> On 1/17/12 11:49 AM, "Kevin E. Thorpe" <kevin.thorpe at utoronto.ca> wrote:
>
>>On 01/17/2012 07:16 AM, Essers, Jonah wrote:
>>> Thanks for the reply. I think more the issue is whether it can be
>>>applied
>>> to cross-sectional data. This I'm not sure. This method is heavily cited
>>> in the New England Journal of Medicine, but thus far I've only seen it
>>> used with longitudinal data.
>>
>>As I recall, the Pencina et al paper does not suggest it cannot be used
>>outside of longitudinal data.  In fact, I don't remember them using
>>longitudinal data at all.  So, unless I'm misunderstanding your
>>question, I think the function in Hmisc (whose name I always forget)
>>should be fine.
>>
>>>
>>> On 1/16/12 10:23 PM, "Kevin E. Thorpe"<kevin.thorpe at utoronto.ca>  wrote:
>>>
>>>> On 01/16/2012 08:10 PM, Essers, Jonah wrote:
>>>>> Greetings,
>>>>>
>>>>> I have generated several ROC curves and would like to compare the
>>>>>AUCs.
>>>>> The data are cross sectional and the outcomes are binary. I am testing
>>>>> which of several models provide the best discrimination. Would it be
>>>>> most
>>>>> appropriate to report AUC with 95% CI's?
>>>>>
>>>>> I have been looking in to the "net reclassification improvement" (see
>>>>> below for reference) but thus far I can only find a version in Hmisc
>>>>> package which requires survival data. Any idea what the best approach
>>>>>is
>>>>> for cross-sectional data?
>>>>
>>>> I believe that the function in Hmisc that does this will also work on
>>>> binary data.
>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS. Evaluating
>>>>>the
>>>>> added predictive ability of a new marker: from area under the ROC
>>>>>curve
>>>>> to
>>>>> reclassification and beyond. Stat Med 2008;27:157-172
>>>>>
>>>>
>>
>>
>>--
>>Kevin E. Thorpe
>>Biostatistician/Trialist,  Applied Health Research Centre (AHRC)
>>Li Ka Shing Knowledge Institute of St. Michael's
>>Assistant Professor, Dalla Lana School of Public Health
>>University of Toronto
>>email: kevin.thorpe at utoronto.ca  Tel: 416.864.5776  Fax: 416.864.3016
>