[R] comparing random forests and classification trees

[jamesmcc]

Greetings tree and forest coders-

I'm interested in comparing randomforests and regression tree/ bagging tree
models. I'd like to propose a basis for doing this, get feedback, and
document this here. I kept it in this thread since that makes sense.

In this case I think it's appropriate to compare the R^2 values as one basic
measure. I'm actually going to compare mean error (ME), mean absolute error
(MAE), root mean squared error (RMSE) as well. This means that I need
estimates from each approach so that I can form residuals. **As I see it,
the important details are in how to set up the models so that I have
comparable estimates, particularly in how the trees/forests are trained and
evaluated.**

For regression/bagging trees, the typical approach for my application is 100
runs of 10-fold CV. In each run all the values are estimated in an
out-of-the-bag sense; each "fold" is estimated while it is withheld from
fitting, thus fit is not inflated. The estimates are then averaged over the
100 runs at each point to get an average simulation and this is used to
calculate residuals and the measures mentioned above. Somewhat more
specifically, the steps are: I fit a model, I prune it via inspection, I
loop 100 times on xpred.rpart(model,xval=10,cp=cp at bottom of cptable from
pruned fit) to generate the 100 runs (bagging is thus performed while
holding the cp criteria fixed?), I average these pointwise, I calculate the
desired stats/quantities for comparison to other models.

For randomForests, I would want to fit the model in a similar way, ie 100
runs of 10-fold CV. I think the 10-fold part is clear, the 100 runs, maybe
less so. To get 10-fold OOB estimates, I set replace=FALSE,
sampsize=.9*nrow(x). Then I get a randomForest with $predicted being the
average OOB estimates over all trees for which each point was OOB. I would
assume that each tree is constructed with a different 10-fold partitioning
of the data set. Thus the number of runs is really more like the number of
trees constructed. If i wanted to be really thorough, I could fit 100 random
forests and get the $predicted for each and then average these pointwise.
But that seems like over kill; isnt that the lesson of plot.randomForest
that as the # of trees goes up the error converges to some limit. (from what
i've seen). 

Thus, my primary concern is in the amount of data used for training and
cross validating the model in an out-of-bag sense; can i meaningfully
compare 10-fold oob estimates sing xpred.rpart to a random forest fit using
90% of the data as sampsize? 

Of secondary concern is the number of bagging trees versus then number of
trees in the random forest. As long as the average estimate error is nearing
some limit with the number of bagging trees I'm using, I think this is all
that matters. So this is more of methodological difference to be retained,
similar to differences in pruning under bagging and random forests, though I
should probably specify the node sizes to be similar for each.

Am I overlooking anything of grave consequence?

Any and all thoughts are welcome. If you are aware of any comparisons of
rpart and randomForests in the literature for any field (for regression) of
which I am ignorant, I would appreciate the tip. I have read over "Newer
Classification and Regression Tree Techniques: Bagging and Random Forests
for Ecological Prediction" by Prasad, Iverson, and Liaw. I may have missed
it, but I did not see discussion of maintaining consistency in the way the
models were trained, though it is a very nice paper overall and contained
many interesting approaches and points. 

Thanks in advance, 

James

-- 
View this message in context: http://www.nabble.com/-R--comparing-random-forests-and-classification-trees-tp8682315p25491934.html
Sent from the R help mailing list archive at Nabble.com.