[R] Neural Network resource

[jude.ryan at ubs.com]

The package AMORE appears to be more flexible, but I got very poor
results using it when I tried to improve the predictive accuracy of a
regression model. I don't understand all the options well enough to be
able to fine tune it to get better predictions. However, using the
nnet() function in package VR gave me decent results and is pretty easy
to use (see the Venables and Ripley book, Modern Applied Statistics with
S, pages 243 to 249, for more details). I tried using package neuralnet
as well but the neural net failed to converge. I could not figure out
how to set the threshold option (or other options) to get the neural net
to converge. I explored package neural as well. Of all these 4 packages,
the nnet() function in package VR worked the best for me.

 

As another R user commented as well, you have too many hidden layers and
too many neurons. In general you do not need more than 1 hidden layer.
One hidden layer is sufficient for the "universal approximator" property
of neural networks to hold true. As you keep adding neurons to the one
hidden layer, the problem becomes more and more non-linear. If you add
too many neurons you will overfit. In general, you do not need to add
more than 10 neurons. The activation function in the hidden layer of
Venables and Ripley's nnet() function is logistic, and you can specify
the activation function in the output layer to be linear using linout =
T in nnet(). Using one hidden layer, and starting with one hidden neuron
and working up to 10 hidden neurons, I built several neural nets (4,000
records) and computed the training MSE. I also computed the validation
MSE on a holdout sample of over 1,000 records. I also started with 2
variables and worked up to 15 variables in a "for" loop, so in all, I
built 140 neural nets using 2 "for" loops, and stored the results in
lists. I arranged my variables in the data frame based on correlations
and partial correlations so that I could easily add variables in a "for"
loop. This was my "crude" attempt to simulate variable selection since,
from what I have seen, neural networks do not have variable selection
methods. In my particular case, neural networks gave me marginally
better results than regression. It all depends on the problem. If the
data has non-linear patterns, neural networks will be better than linear
regression.

 

My code is below. You can modify it to suit your needs if you find it
useful. There are probably lines in the code that are redundant which
can be deleted.

 

HTH.

 

Jude Ryan

 

My code:

 

# set order in data frame train2 based on correlations and partial
correlations

train2 <- train[, c(5,27,19,20,25,26,4,9,3,10,16,6,2,14,21,28)]

dim(train2)

names(train2)

library(nnet)

# skip = T

# train 10 neural networks in a loop and find the one with the minimum
test and validation error

# create various lists to store the results of the neural network
running in two for loops

# The Column List is for the outer for loop, which loops over variables

# The Row List is for the inner for loop, which loops over number of
neurons in the hidden layer

col_nn <- list()  # stores the results of nnet() over variables - outer
loop

row_nn <- list()  # stores the results of nnet() over neurons - inner
loop

col_mse <- list()

# row_mse <- list() # not needed because nn.mse is a data frame with
rows

col_sum <- list()

row_sum <- list()

col_vars <- list()

row_vars <- list()

col_wts <- list()

row_wts <- list()

df_dim <- dim(train2)

df_dim[2]  # number of variables

df_dim[2] - 1

num_of_neurons <- 10

# build data frame to store results of neural net for each run

nn.mse <- data.frame(Train_MSE=seq(1:num_of_neurons),
Valid_MSE=seq(1:num_of_neurons))

# open log file and redirect output to log file

sink("D:\\XXX\\YYY\\ Programs\\Neural_Network_v8_VR_log.txt")

# outer loop - loop over variables

for (i in 3:df_dim[2]) {  # df_dim[2]

  # inner loop - loop over number of hidden neurons

  for (j in 1:num_of_neurons) { # upto 10 neurons in the hidden layer

    # need to create a new data frame with just the predictor/input
variables needed

    train3 <- train2[,c(1:i)]

    coreaff.nn <- nnet(dep_var ~ ., train3, size = j, decay = 1e-3,
linout = T, skip = T, maxit = 1000, Hess = T)

    # row_vars[[j]] <- coreaff.nn$call # not what we want

    # row_vars[[j]] <- names(train3)[c(2:i)] # not needed in inner loop
- same number of variables for all neurons

    row_sum[[j]] <- summary(coreaff.nn)

    row_wts[[j]] <- coreaff.nn$wts

    rownames(nn.mse)[j] <- paste("H", j, sep="")

    nn.mse[j, "Train_MSE"] <- mean((train3$dep_var -
predict(coreaff.nn))^2)

    nn.mse[j, "Valid_MSE"] <- mean((valid$dep_var - predict(coreaff.nn,
valid))^2)

  }

  col_vars[[i-2]] <- names(train3)[c(2:i)]

  col_sum[[i-2]] <- row_sum

  col_wts[[i-2]] <- row_wts

  col_mse[[i-2]] <- nn.mse

}

# cbind(col_vars[1],col_vars[2])

col_vars

col_sum

col_wts

sink()

cbind(col_mse[[1]],col_mse[[2]],col_mse[[3]],col_mse[[4]],col_mse[[5]],c
ol_mse[[6]],col_mse[[7]],

 
col_mse[[8]],col_mse[[9]],col_mse[[10]],col_mse[[11]],col_mse[[12]],col_
mse[[13]],col_mse[[14]])

# build a neural network with the same 9 variables as in regression

pred <- list()

summaries <- list()

wts <- list()

 

Indrajit wrote:

 

You are right there is a pdf file which describes the function. But let
tell you where I am coming from.

 

Just to test if a neural network will work better than a ordinary least
square regression, I created a dataset with one dependent variable and 6
other independent variables. Now I had deliberately created the dataset
in such manner that we have an excellent regression model. Eg: Y = b0 +
b1*x1 + b2*x2 + b3*x3.. + b6*x6 + e

where e is normal random variable. Naturally any statistical analysis
system running regression would easily predict the values of b1, b2, b3,
..., b6 with around 30-40 observations.

 

I fed this data into a Neural network (3 hidden layers with 6 neurons in
each layer) and trained the network. When I passed the input dataset and
tried to get the predictions, all the predicted values were identical!
This confused me a bit and was wondering whether my understanding of the
Neural Network was wrong.

 

Have you ever faced anything like it?

 

Regards,

Indrajit

 

 

___________________________________________
Jude Ryan
Director, Client Analytical Services
Strategy & Business Development
UBS Financial Services Inc.
1200 Harbor Boulevard, 4th Floor
Weehawken, NJ 07086-6791
Tel. 201-352-1935
Fax 201-272-2914
Email: jude.ryan at ubs.com



-------------- next part --------------
Please do not transmit orders or instructions regarding a UBS 
account electronically, including but not limited to e-mail, 
fax, text or instant messaging. The information provided in 
this e-mail or any attachments is not an official transaction 
confirmation or account statement. For your protection, do not 
include account numbers, Social Security numbers, credit card 
numbers, passwords or other non-public information in your e-mail. 
Because the information contained in this message may be privileged, 
confidential, proprietary or otherwise protected from disclosure, 
please notify us immediately by replying to this message and 
deleting it from your computer if you have received this 
communication in error. Thank you. 

UBS Financial Services Inc. 
UBS International Inc. 
UBS Financial Services Incorporated of Puerto Rico 
UBS AG

 
UBS reserves the right to retain all messages. Messages are protected
and accessed only in legally justified cases.