3
$\begingroup$

I have a Pandas dataframe X(20346, 4116). All independent columns have binary variable as 0 or 1. Whereas dependent column has continuous variable. When I execute below code using the scikit-learn library, it takes for ever and I am not getting any result.

X_train, X_test, y_train, y_test = train_test_split(X13, y, train_size=0.8)

rf = RandomForestRegressor(n_estimators=100,  max_features='auto', oob_score=True,
                            n_jobs=1, verbose=0)
rf.fit(X_train, y_train) 

I am using this for feature selection. In case of less number of rows I can get result. But when number of rows go up then this problem starts happening. Do you have any idea how to make it work?

$\endgroup$
2
$\begingroup$

Some of the settings are likely too aggressive for your machine:

As others have pointed out, n_jobs might help by taking advantage of openMP threading that is likely available on a multicore processor.

But lets also look at some other parameters from the documentation:

RandomForestRegressor(n_estimators=10, criterion='mse', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features='auto', max_leaf_nodes=None, bootstrap=True, oob_score=False, n_jobs=1, random_state=None, verbose=0, warm_start=False)
  • Why don't you set n_estimators to 10 and then slowly ramp it up.
  • max_depth should probably be set both so you trees aren't too computationally intensive and so you don't overfit your model (are you cross validating?)
  • You have set max_features to auto which specifies that it use n_features, but you have not set n_features. You could consider turning this down to some percentage of your full feature set.

In general you need to cross validate your model and tinker with the parameters in order to find something that runs on your box in finite time using the available memory and performs adequately. You could also consider tuning and cross validating a simpler decision tree and then using that insight to tweak the parameters of the random forest.

Sample the data:

Big datasets often require lots of computing power. I sometimes drastically reduce the size of my training data while I am zeroing in on the best model and then time the scaling as I increase the dataset. This will give you a better idea of how much time you should expect for the full problem.

Other models:

Finally, maybe you have a problem for which 'bootstrap aggregation' is a bad fit. Maybe another type of aggregation would work better like 'boosting' in which case a 'random jungle' or 'ada-boost' algo might work better. Or perhaps you might want to move to a computationally less intense method that doesn't require aggregation like a 'support vector regressor'. Scikit-learn has a great benefit in that it is very simple to swap out different models and try different pipelines.

Cloud Computing:

I'm not sure what box or laptop you are running this on, but it might just be easier to spin up an AWS (Google/MS/IBM) instance and run this on a well-powered box in the cloud. The AWS spot market is crazy affordable over the weekends or in the middle of the night, so you can work on some very powerful 8 core machines for $.50 per hour, so a 24 hour job is only $12.

Hope this helps!

| improve this answer | |
$\endgroup$
0
$\begingroup$

Have you tried increasing n_jobs or setting oob_score to False? I am not sure what the logic is in sklearn to default max_features to auto (=n_features). Using sqrt is common in random forest and should gain efficiency.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.