This is an issue for all Data Scientists who have worked with this stack:

  • python
  • scikit-learn
  • scipy-stats
  • matplotlib
  • etc.

We are looking for ways to have a project already implemented in the aforementioned stack scale for very large datasets by doing the minimum amount of work

Counter examples would be to rewrite everything in Tensorflow framework or use industry tools that are unrelated with Python.

  • 2
    $\begingroup$ what kind of things have you tried? What about moving to AWS, or some other cloud service that provides beefier computers? $\endgroup$ Oct 19, 2017 at 15:04
  • 2
    $\begingroup$ pySpark is the most obvious option, but you'll have to use MLLib instead of scikit-learn. $\endgroup$
    – Emre
    Oct 19, 2017 at 16:30
  • $\begingroup$ What algorithms of scikit-learn are you using (and why)? $\endgroup$
    – stmax
    Oct 19, 2017 at 18:41
  • $\begingroup$ This is pretty vague. If you just need to retrain the model on a larger dataset then try to do some out-of-core processing, run it on a server and have an SMTP handler email you when it's done. Once there you can pickle the model and from there predictions shouldn't be to bad. However, if you need to constantly train the model and each run is larger, you probably have to move from sklearn and into a more distributive approach. $\endgroup$
    – Tophat
    Dec 18, 2017 at 17:16

2 Answers 2


The easiest way (depending on the scale we're talking about) is to set n_jobs=-1 for algorithms that support parallelization (e.g. random forest, cross validation, grid search). This will take advantage of all the cores on your machine. If that's not good enough, you should probably move to spark.


You generally don't. Scikit-learn is primarily aimed to help new data scientists quickly get comfortable with data science

That being said, some strategies for scaling are discussed here: http://scikit-learn.org/stable/modules/scaling_strategies.html

This includes using out-of-core models, reducing data size with PCA, and various incremental learners

Besides that, your best bet is to use a beefier computer

Also, remember that once a model is trained, it can be pickled and shared. Training/testing is usually the time/cpu consuming process. So, once you have a model, you should be able to implement it on machines less beefy than the train/test machine


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.