[R] Comparing differences in AUC from 2 different models

[Monica Pisica]

Hi,

I would like to compare differences in AUC from 2 different models, glm and gam for predicting presence / absence. I know that in theory the model with a higher AUC is better, but what I am interested in is if statistically the increase in AUC from the glm model to the gam model is significant. I also read quite extensive discussions on the list about ROC and AUC but I still didn't find my answer.

To calculate the AUC and plot the ROC I used the package PresenceAbsence. The help file for auc() says: " The standard errors from auc are only valid for comparing an individual model to random assignment (i.e. AUC=.5). To compare two models to each other it is necessary to account for correlation due to the fact that they use the same test set. If you are interested in pair wise model comparisons see the Splus ROC library from Mayo clinic. auc is a much simpler function than what is available from the Splus ROC library from Mayo clinic."

I did download this library but I don't have access to S-PLUS and even if supposedly the code is very similar between S-PLUS and R I still don't quite understand what is going on because I am a little bit confused what some parameters represent …. For example "markers" and "status", although I think "status" represent my original data (all coded 0 and 1) and "markers" might be the probabilities obtained from my 2 models. The confusion may also steam from the fact that I don't have a medical or biological training and maybe "markers" and "status" do have a special meaning for these 2 disciplines. 

I will really appreciate if you can help in finding a way to compare differences in AUC.

Thanks,

Monica



_________________________________________________________________


_WL_Refresh_messenger_video_072008