Thanks to all those who have replied to my query. I have decided do a thorough reading on this subject and try to seek out a proper solution. I will stick to the nnet package as mentioned by Jude and try and compare results with other neural network software if possible.

Regards,
Indrajit
 




________________________________
From: "jude.ryan@ubs.com" <jude.ryan@ubs.com>

Sent: Thursday, May 28, 2009 10:49:36 PM
Subject: Re: [R] Neural Network resource


The package AMORE appears to be more flexible, but I got very poor results using it when I tried to improve the predictive accuracy of a regression model. I don't understand all the options well enough to be able to fine tune it to get better predictions. However, using the nnet() function in package VR gave me decent results and is pretty easy to use (see the Venables and Ripley book, Modern Applied Statistics with S, pages 243 to 249, for more details). I tried using package neuralnet as well but the neural net failed to converge. I could not figure out how to set the threshold option (or other options) to get the neural net to converge. I explored package neural as well. Of all these 4 packages, the nnet() function in package VR worked the best for me.
 
As another R user commented as well, you have too many hidden layers and too many neurons. In general you do not need more than 1 hidden layer. One hidden layer is sufficient for the "universal approximator" property of neural networks to hold true. As you keep adding neurons to the one hidden layer, the problem becomes more and more non-linear. If you add too many neurons you will overfit. In general, you do not need to add more than 10 neurons. The activation function in the hidden layer of Venables and Ripley's nnet() function is logistic, and you can specify the activation function in the output layer to be linear using linout = T in nnet(). Using one hidden layer, and starting with one hidden neuron and working up to 10 hidden neurons, I built several neural nets (4,000 records) and computed the training MSE. I also computed the validation MSE on a holdout sample of over 1,000 records. I also started with 2 variables and worked up to 15 variables in
 a "for" loop, so in all, I built 140 neural nets using 2 "for" loops, and stored the results in lists. I arranged my variables in the data frame based on correlations and partial correlations so that I could easily add variables in a "for" loop. This was my "crude" attempt to simulate variable selection since, from what I have seen, neural networks do not have variable selection methods. In my particular case, neural networks gave me marginally better results than regression. It all depends on the problem. If the data has non-linear patterns, neural networks will be better than linear regression.
 
My code is below. You can modify it to suit your needs if you find it useful. There are probably lines in the code that are redundant which can be deleted.
 
HTH.
 
Jude Ryan
 
My code:
 
# set order in data frame train2 based on correlations and partial correlations
train2 <- train[, c(5,27,19,20,25,26,4,9,3,10,16,6,2,14,21,28)]
dim(train2)
names(train2)
library(nnet)
# skip = T
# train 10 neural networks in a loop and find the one with the minimum test and validation error
# create various lists to store the results of the neural network running in two for loops
# The Column List is for the outer for loop, which loops over variables
# The Row List is for the inner for loop, which loops over number of neurons in the hidden layer
col_nn <- list()  # stores the results of nnet() over variables - outer loop
row_nn <- list()  # stores the results of nnet() over neurons - inner loop
col_mse <- list()
# row_mse <- list() # not needed because nn.mse is a data frame with rows
col_sum <- list()
row_sum <- list()
col_vars <- list()
row_vars <- list()
col_wts <- list()
row_wts <- list()
df_dim <- dim(train2)
df_dim[2]  # number of variables
df_dim[2] - 1
num_of_neurons <- 10
# build data frame to store results of neural net for each run
nn.mse <- data.frame(Train_MSE=seq(1:num_of_neurons), Valid_MSE=seq(1:num_of_neurons))
# open log file and redirect output to log file
sink("D:\\XXX\\YYY\\ Programs\\Neural_Network_v8_VR_log.txt")
# outer loop - loop over variables
for (i in 3:df_dim[2]) {  # df_dim[2]
  # inner loop - loop over number of hidden neurons
  for (j in 1:num_of_neurons) { # upto 10 neurons in the hidden layer
    # need to create a new data frame with just the predictor/input variables needed
    train3 <- train2[,c(1:i)]
    coreaff.nn <- nnet(dep_var ~ ., train3, size = j, decay = 1e-3, linout = T, skip = T, maxit = 1000, Hess = T)
    # row_vars[[j]] <- coreaff.nn$call # not what we want
    # row_vars[[j]] <- names(train3)[c(2:i)] # not needed in inner loop - same number of variables for all neurons
    row_sum[[j]] <- summary(coreaff.nn)
    row_wts[[j]] <- coreaff.nn$wts
    rownames(nn.mse)[j] <- paste("H", j, sep="")
    nn.mse[j, "Train_MSE"] <- mean((train3$dep_var - predict(coreaff.nn))^2)
    nn.mse[j, "Valid_MSE"] <- mean((valid$dep_var - predict(coreaff.nn, valid))^2)
  }
  col_vars[[i-2]] <- names(train3)[c(2:i)]
  col_sum[[i-2]] <- row_sum
  col_wts[[i-2]] <- row_wts
  col_mse[[i-2]] <- nn.mse
}
# cbind(col_vars[1],col_vars[2])
col_vars
col_sum
col_wts
sink()
cbind(col_mse[[1]],col_mse[[2]],col_mse[[3]],col_mse[[4]],col_mse[[5]],col_mse[[6]],col_mse[[7]],
      col_mse[[8]],col_mse[[9]],col_mse[[10]],col_mse[[11]],col_mse[[12]],col_mse[[13]],col_mse[[14]])
# build a neural network with the same 9 variables as in regression
pred <- list()
summaries <- list()
wts <- list()
 
Indrajit wrote:
 
You are right there is a pdf file which describes the function. But let tell you where I am coming from.
 
Just to test if a neural network will work better than a ordinary least square regression, I created a dataset with one dependent variable and 6 other independent variables. Now I had deliberately created the dataset in such manner that we have an excellent regression model. Eg: Y = b0 + b1*x1 + b2*x2 + b3*x3.. + b6*x6 + e
where e is normal random variable. Naturally any statistical analysis system running regression would easily predict the values of b1, b2, b3, ..., b6 with around 30-40 observations.
 
I fed this data into a Neural network (3 hidden layers with 6 neurons in each layer) and trained the network. When I passed the input dataset and tried to get the predictions, all the predicted values were identical! This confused me a bit and was wondering whether my understanding of the Neural Network was wrong.
 
Have you ever faced anything like it?
 
Regards,
Indrajit
 
 
___________________________________________
Jude Ryan
Director, Client Analytical Services
Strategy & Business Development
UBS Financial Services Inc.
1200 Harbor Boulevard , 4th Floor
Weehawken , NJ 07086-6791
Tel. 201-352-1935
Fax 201-272-2914
Email: jude.ryan@ubs.com


      
	[[alternative HTML version deleted]]


