[R] memory problems when combining randomForests

[Eleni Rapsomaniki]

Hi Andy,

> > I get different order of importance for my variables depending on their
order in the training data.

Perhaps answering my own question, the change in importance rankings could be
attributed to the fact that before passing my data to randomForest I impute the
missing values randomly (using the combined distributions of pos+neg), so the
data seen by RF is slightly different. Then combining this with the fact that
RF chooses data randomly it makes sense to see different rankings.

In a previous thread regarding simplifying variables:
http://thread.gmane.org/gmane.comp.lang.r.general/6989/focus=6993

you say:
"The basic problem is that when you select important variables by RF and then
re-run RF with those variables, the OOB error rate become biased downward.
As you iterate more times, the "overfitting" becomes more and more severe
(in the sense that, the OOB error rate will keep decreasing while error rate
on an independent test set will be flat or increases)" 

But if every time you remove a variable you pass some test data (ie data not
used to train the model) and base the performance of the new, reduced model on
the error rate on the confusion matrix for the test data, then this
"overfitting" should not be an issue, right?  (unless of course you were
referring to unsupervised learning).

Best regards
Eleni Rapsomaniki
Birkbeck College, UK