[R] nnet classification accuracy vs. other models

[Prof Brian Ripley]

On Sun, 14 Mar 2004, Albedo wrote:

> The only thing that I could have done wrong with nnet
> (that I 
> can think of) is not enough nuerons in hidden layer,
> but then
> again this is actually limited by my computer memory.

Perhaps you had too many, not too few?  Perhaps you didn't choose the 
weight decay correctly?  Perhaps you made an error in the code you used?
Who knows?  You haven't told us anything useful about what you did ....

> However, I did estimate the error a little bit
> different - I have
> enough data for test set, which I used for classification
> accuracy estimation only.

So you may have the tree results wrong, instead?

I suggest that you seek expert statistical help from a paid consultant:  
these are not R questions and you seem to need a face-to-face dialogue
even to help you ask answerable questions.  Your basic hypothesis is
wrong:  there is good reason to expect nnet to outperform tree models
(even after bagging the latter, and `bagging' is not a model), *in the
hands of a competent user*.


>  Edgar Acuna <edgar at cs.uprm.edu>:
> 
> > I think that you are using nnet incorrectly. I have
> compared several
> > classifiers (including that ones that you mention in
> your e-mail) on the
> > same dataset and I have never found more of a 20% of
> difference in the
> > missclassification error. Of course, I estimated the
> misclassification
> > error by cross validation.
> >
> > Regards
> > Edgar Acuna
> > UPR-MATH
> >
> > On Sat, 13 Mar 2004, Albedo wrote:
> >
> > > I was wandering if anybody ever tried to compare
> the classification
> > > accuracy of nnet to other (rpart, tree, bagging)
> models. From what I
> > > know, there is no reason to expect a significant
> difference in
> > > classification accuracy between these models, yet
> in my particular case
> > > I get about 10% error rate for tree, rpart and
> bagging model and 80%
> > > error rate for nnet, applied to the same data.

-- 
Brian D. Ripley,                  ripley at stats.ox.ac.uk
Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
University of Oxford,             Tel:  +44 1865 272861 (self)
1 South Parks Road,                     +44 1865 272866 (PA)
Oxford OX1 3TG, UK                Fax:  +44 1865 272595