[R] net classification improvement?

[Kevin E. Thorpe]

On 01/17/2012 11:55 AM, Essers, Jonah wrote:
> Actually, I don't think I made myself clear and I wrote this late last
> night....Sorry. More the issue is that the raw model predictions (from 0
> to 1) have no inherent clinical value to them. I.e. They aren't "risk of
> disease" or "risk of outcome". They are raw scores that are specific to
> each model and are meant to discriminate one disease from another disease.
> Trying to compare models is impossible because the NRI requires cutoff
> values.  The cutoffs are different for each model.
>
> So, as I've done more reading, it appears the the IRI--Integrated
> Discrimination Improvement Index--which is naïve to cutoff values--may be
> more what I'm looking for. Does this make sense? I guess I just need a
> sanity check.

Yes, the IRI makes sense to me.

>
> I have been toying with the PredictABEL package and this seems to like my
> data inputs just fine and relies on HMISC and ROCR, both packages I know
> well.
>
> Thanks
> jonah
>
> On 1/17/12 11:49 AM, "Kevin E. Thorpe"<kevin.thorpe at utoronto.ca>  wrote:
>
>> On 01/17/2012 07:16 AM, Essers, Jonah wrote:
>>> Thanks for the reply. I think more the issue is whether it can be
>>> applied
>>> to cross-sectional data. This I'm not sure. This method is heavily cited
>>> in the New England Journal of Medicine, but thus far I've only seen it
>>> used with longitudinal data.
>>
>> As I recall, the Pencina et al paper does not suggest it cannot be used
>> outside of longitudinal data.  In fact, I don't remember them using
>> longitudinal data at all.  So, unless I'm misunderstanding your
>> question, I think the function in Hmisc (whose name I always forget)
>> should be fine.
>>
>>>
>>> On 1/16/12 10:23 PM, "Kevin E. Thorpe"<kevin.thorpe at utoronto.ca>   wrote:
>>>
>>>> On 01/16/2012 08:10 PM, Essers, Jonah wrote:
>>>>> Greetings,
>>>>>
>>>>> I have generated several ROC curves and would like to compare the
>>>>> AUCs.
>>>>> The data are cross sectional and the outcomes are binary. I am testing
>>>>> which of several models provide the best discrimination. Would it be
>>>>> most
>>>>> appropriate to report AUC with 95% CI's?
>>>>>
>>>>> I have been looking in to the "net reclassification improvement" (see
>>>>> below for reference) but thus far I can only find a version in Hmisc
>>>>> package which requires survival data. Any idea what the best approach
>>>>> is
>>>>> for cross-sectional data?
>>>>
>>>> I believe that the function in Hmisc that does this will also work on
>>>> binary data.
>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>> Pencina MJ, D'Agostino RB Sr, D'Agostino RB Jr, Vasan RS. Evaluating
>>>>> the
>>>>> added predictive ability of a new marker: from area under the ROC
>>>>> curve
>>>>> to
>>>>> reclassification and beyond. Stat Med 2008;27:157-172
>>>>>
>>>>
>>


-- 
Kevin E. Thorpe
Biostatistician/Trialist,  Applied Health Research Centre (AHRC)
Li Ka Shing Knowledge Institute of St. Michael's
Assistant Professor, Dalla Lana School of Public Health
University of Toronto
email: kevin.thorpe at utoronto.ca  Tel: 416.864.5776  Fax: 416.864.3016