0
$\begingroup$

I am trying to build a random forest model in R (RStudio). My training dataset has around 2 million rows and 38 variables. When I tested 5000 rows from this dataset I was able to build the random forest but when I run on the whole dataset I get the following error:

Error in randomForest.default(m, y, ...) : long vectors (argument 24) are not supported in .C

Can anyone please suggest, apart from removing the number of rows, how can I fix this? Can I run multiple random forests and then combine them into one? If yes, can someone please recommend how can I try this?

Many thanks in advance.

$\endgroup$
0
$\begingroup$

I'm very surprised you're running into this error with only 2mil rows and 38 variables. I would encourage you to have a go doing this in python using SKLearn and see if you run into the same issue. More generally though, if you have too much data (imagine you had 200 million rows), the right thing to do would not be to build multiple forests, but to build each tree with a smaller fraction of the data.

I don't have a suggestion for where to find such an implementation (there might be some way of combining sklearn and dask, I've seen such things for XGBoost), but the general principle would be that you wouldn't hold all of your data in memory, you'd read a subset of it from disk, train a tree, read a new subset, etc.

$\endgroup$
  • $\begingroup$ Thanks a lot for your answer. Next time I will try this Python. This time I fixed it by using mtry on small dataset to find the number of trees good enough and then ran the whole model with ntrees =300 and mtry=12. It worked this time. $\endgroup$ – T.H. Sep 28 '18 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.