I have a big dataset (1million x 50) to which I want to predict a particular class. I have thought of segregating the dataset in batches of 20k. And then train a classifier (lets say random forest or a basic SVM). How do I then improve that classifier by providing it with additional dataset. In other words, how do I preserve the random forests created in iteration i and use as starting model in interation i+1 to improve the model in python?

  • $\begingroup$ some models are retrainable, RF is not (but you can add fresh trees). Ask your advisor for papers to read on basic machine learning models so you can better understand which ones are retrainable. $\endgroup$ – Mohammad Athar Sep 5 '18 at 19:45

The answer depends on your motivation for breaking the data into blocks. It could range from 'use online training' to 'use ensemble methods' to 'don't break the data into blocks'. Here are a few options to consider.

You could train multiple models on different subsets of the data, then ensemble them, as @Hobbes suggested. This would let you train the models in parallel on different machines. The function learned would be different than training a single model on the full data set.

If using a linear SVM, you can simply continue training the model. Say you train on the first 20k points, then receive an additional 20k points. Initialize the optimization using the first model, then train using the full set of 40k points. Online methods like stochastic gradient descent are a good way to train SVMs on large data sets. SGD processes a single point at a time, so dividing the data into blocks isn't necessary. If the full dataset is available, you'd probably get faster convergence by sweeping through it all instead of breaking it into blocks.

I'd venture a guess that you're not directly using a kernelized SVM on 1M data points. But, you could approximate one by using the kernel to perform a feature space mapping, then train a linear SVM in the feature space. For example, the Nyström technique approximates the mapping using a random subsample of data points. One option would be to learn the feature space mapping using all data, then incrementally train the linear SVM. An alternative would be to learn the mapping using only the first block of data. This is valid if all blocks are equally representative of the full data set and adding new data wouldn't change the mapping.

The case isn't so straightforward for random forests because the individual trees choose hierarchical splits in a greedy manner. The splits chosen for the initial 20k points may not match what would have been chosen had the full 40k points been available. But, it's not easy to go back and revise them. However, a number of papers have proposed online variants of random forests that could be used in this setting. A search for 'incremental' or 'online' random forests should turn up the relevant results.


The simplest, and likely best, way is to just rebuild the model every time you add more data.

If you want to build on existing models, one way is to average multiple random forest models. So you build a model on 20k, test it, then decide to build on it by building another random forest model on a separate set of 20k and averaging the model parameter results with the previous model. This is basically a poorly described ensemble method. Check out this: http://scikit-learn.org/stable/modules/ensemble.html.


I think using the ensemble method would be the best. It will add more accuracy and reduce the bias.

Generally, these 8 steps can help:

  1. Add more data
  2. Treat missing and Outlier values
  3. Feature Engineering
  4. Feature Selection
  5. Multiple algorithms
  6. Algorithm Tuning
  7. Ensemble methods
  8. Cross Validation



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.