[R] Accuracy/Goodness of fit of nnet

[Claudia Beleites]

Raji,

you first need to tell us what kind of accuracy you mean.

The term accuracy has different meanings in different areas of science.
However, in classification it usually refers to something along the line number 
of correctly predicted samples / total number of samples (possibly weighted 
according to the number of samples per class).

Procedures:
You can calculate that for different "types" of test samples:

- prediction of the training samples gives you a goodness of fit. If you have 
(too) many variates you have in your model, this measure is close to useless. 
Useless, because most people are not interested in the goodness of fit anyways 
but want to know the performance for new samples.

- prediction of unknown (statistically independent) samples: this is usually 
what is of interest. You may use resampling schemes (out-of-bootstrap & Co., 
(iterated) cross validation).
There's package boot (though I never used it as it does not properly fit my data)

- Resampling schemes usually cannot tell you the performance for /future/ 
samples: for that you need a test set that is acquired later (and as close as 
possible to the real data to predict).
You need to do this if you want to take into account things like instrument 
drift etc.

There's tons of literature around, what to do depends somewhat on your field. I 
can point you to chemometric literature.

Calculating:
- package ROCR calculates all sorts of classifier performance measures for 
binary classification
.
- I'm developing a package that gives performance measures directly for 
continuous predictions (such as predict.mulitnom with type = "probs"). You are 
welcome to be a test user: just let me know if you want to try it out.


Hope that helps,

Claudia




On 10/21/2010 05:37 AM, Raji wrote:
>
> Hi R-Helpers , am working on nnet package.Multinom() has an option for
> finding the goodness of fit by giving the AIC value. Does nnet also gives
> some value to determine the accuracy. If not, can you guide me with some
> procedure to figure out the accuracy/goodness of fit of nnet model?
>
> Thanks in advance.


-- 
Claudia Beleites
Dipartimento dei Materiali e delle Risorse Naturali
Università degli Studi di Trieste
Via Alfonso Valerio 6/a
I-34127 Trieste

phone: +39 0 40 5 58-37 68
email: cbeleites at units.it