Thanks Mr. Ellison,

Your remark helped solve my error table problem.

However, I found a new one.

Now that I have my error tables, i realised that it is no good statistical 
practise to validate a model, based one one error table.

So i should use a tool like K-fold CV.


ex:     binary_model <- glm (y_binary~ x_value, family = 
binomial,data=dataset)

        cv.binary(binary_model,rand=NULL, nfolds=1000, print.details=TRUE)


This is no problem for the binary model, for the odds model this is not 
the case.

Do you know a tool that can do this, or perhapes a way to implement it in 
a monte carlo simulation?

(i added the way i solved the error table problem below)


Kind regards,

Tom.



>ERROR TABLE DILEMMA

>For a binary model there is no problem (here y has an outcome of 0 or 1)

>

>ex:    pred_binary_model=(expit(predict(binary_model,tsample))>P)

>       table_binary_model=table(pred_binary_model,tsample[,2])

> 
TER_binary_model=sum(diag(table_binary_model[,]))/sum(table_binary_model)

>

>       (error table1)

>       pred_binary_model       0   1

>                       FALSE   28 95

>                       TRUE    4 114

>       [1] 0.5892116 --> of correct classified cases

>

>Here there are 28 + 114 correctly predicted test cases, this results in 
58.9% correct classified cases.

>A few more misclassified cases does not result in big departures from 
this 58.9%.

>

>When i preform this on categorical data, i have to use frequency tables.

>This predicts the number of successes and the number of failures, per 
interval.(odds per interval)

>So the error table does contain an outcome of odds for every given 
interval.

>

>ex: (error table2)

>       oddsPt

>       pred_percent_model      0.00 0.16 0.37 0.84

>                       0.05    1       0   0   0

>                       0.16    0       1   0   0

>                       0.34    0       0   1   0
 
>                       0.78    0       0   0   1

>       [1] 1 --> of correct classified cases

>

>As you can see, one misclassification will take disastrous proportions. 
(~25% difference)

>The output of error table2 is interpretable, but it is not ideal, and 
oversensitive to misclassification.

>

>I was able to solve this later problem by extracting the model 
coefficients, and then using them in a function.

>Based on this function, i was able to write an error table equal to table 
1.

 


Disclaimer: click here
	[[alternative HTML version deleted]]

