[R] nnet classification accuracy vs. other models

[Philippe Grosjean]

I did a comparison of 15 different methods with a particularly difficult
real dataset. I tried to fine-tune all methods the best as I could. I must
admit that fine-tuning of nnet was by far the longest process, and I am
pretty sure it is still not optimal! I did 100 replicates of random
selection of 2/3 of the data for the training. The rest being used for
estimating accuracy (I could have done a n-fold cross-validation as well) of
the methods. (I got the timing too for the methods, because it is an
important factor in my application). The results are presented in the
attached graph. Here is the legend of the labels:
- lda = linear discriminant analysis,
- qda = quadratic discriminant analysis,
- mda = mixture discriminant analysis,
- fda = flexible discriminant analysis,
- knn = k-nearest neighbourg,
- lvq = learning vector quantization,
- tree = tree method (from tree package),
- rpart = tree method (from rpart package),
- bagg = bagging (from ipred),
- db.l = double bagging with lda (ipred),
- db.k = double bagging with knn(ipred),
- rfor = random forest,
- svm = support vector machine,
- nnet = neural network,
- dvf = discriminant vector forest (a new method I have set up).

You see that, if most median values for total accuracy are around between
77% and 85%, there are a few methods that are a little bit less performant
(tree, rpart, svm and nnet). However, a method like svm (from package e1071)
is usually performing better than that and compares often favorably with
other machine learning techniques. So, I checked with its author (David
Meyer) if it was possible to improve these results, and if I did something
wrong. He did not found a way to improve it much more. However, I did not
the same exercice for nnet.

Anyway, this shows that some differences between methods could be expected
and results are not always as expected (that is, application of these
methods on real data do not always match the results obtained with the
various benchmark artificial data often used to compare these methods)!

Best,

Philippe Grosjean

.......................................................<?}))><....
 ) ) ) ) )
( ( ( ( (   Prof. Philippe Grosjean
\  ___   )
 \/ECO\ (   Numerical Ecology of Aquatic Systems
 /\___/  )  Mons-Hainaut University, Pentagone
/ ___  /(   8, Av. du Champ de Mars, 7000 Mons, Belgium
 /NUM\/  )
 \___/\ (   phone: + 32.65.37.34.97, fax: + 32.65.37.33.12
       \ )  email: Philippe.Grosjean at umh.ac.be
 ) ) ) ) )  SciViews project coordinator (http://www.sciviews.org)
( ( ( ( (
...................................................................


-----Original Message-----
From: r-help-bounces at stat.math.ethz.ch
[mailto:r-help-bounces at stat.math.ethz.ch]On Behalf Of Christian Hennig
Sent: Monday, 15 March, 2004 10:19
To: Albedo
Cc: s-news at lists.biostat.wustl.edu; r-help at stat.math.ethz.ch
Subject: Re: [R] nnet classification accuracy vs. other models


My experience is that nnet needs a lot of tuning, not only in terms of
numbers of layers, but also in terms of the other parameters. My first
results where I kept very much of the default parameter values with nnet
have been very bad, as bad as you say. (But as Brian Ripley already wrote,
it's not straight forward to say via the net how to do it better.)

Apart from that, such a large difference of classification accuracy
between different methods is strange, but possible in principle.
Very different structures of data exist (which means again that nobody can
assess your problem without knowing the data).

Christian

On Sat, 13 Mar 2004, Albedo wrote:

> I was wandering if anybody ever tried to compare the classification
> accuracy of nnet to other (rpart, tree, bagging) models. From what I
> know, there is no reason to expect a significant difference in
> classification accuracy between these models, yet in my particular case
> I get about 10% error rate for tree, rpart and bagging model and 80%
> error rate for nnet, applied to the same data.
>
> Thanks.
>
> ______________________________________________
> R-help at stat.math.ethz.ch mailing list
> https://www.stat.math.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html
>

***********************************************************************
Christian Hennig
Fachbereich Mathematik-SPST/ZMS, Universitaet Hamburg
hennig at math.uni-hamburg.de, http://www.math.uni-hamburg.de/home/hennig/
#######################################################################
ich empfehle www.boag-online.de

______________________________________________
R-help at stat.math.ethz.ch mailing list
https://www.stat.math.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide!
http://www.R-project.org/posting-guide.html

-------------- next part --------------
A non-text attachment was scrubbed...
Name: compalm.pdf
Type: application/pdf
Size: 80850 bytes
Desc: not available
Url : https://stat.ethz.ch/pipermail/r-help/attachments/20040315/026c4fbf/compalm.pdf