# What is the easiest way to scale a data science project based on scikit stack?

This is an issue for all Data Scientists who have worked with this stack:

• python
• scikit-learn
• scipy-stats
• matplotlib
• etc.

We are looking for ways to have a project already implemented in the aforementioned stack scale for very large datasets by doing the minimum amount of work

Counter examples would be to rewrite everything in Tensorflow framework or use industry tools that are unrelated with Python.

• what kind of things have you tried? What about moving to AWS, or some other cloud service that provides beefier computers? Oct 19, 2017 at 15:04
• pySpark is the most obvious option, but you'll have to use MLLib instead of scikit-learn.
– Emre
Oct 19, 2017 at 16:30
• What algorithms of scikit-learn are you using (and why)? Oct 19, 2017 at 18:41
• This is pretty vague. If you just need to retrain the model on a larger dataset then try to do some out-of-core processing, run it on a server and have an SMTP handler email you when it's done. Once there you can pickle the model and from there predictions shouldn't be to bad. However, if you need to constantly train the model and each run is larger, you probably have to move from sklearn and into a more distributive approach. Dec 18, 2017 at 17:16

The easiest way (depending on the scale we're talking about) is to set n_jobs=-1 for algorithms that support parallelization (e.g. random forest, cross validation, grid search). This will take advantage of all the cores on your machine. If that's not good enough, you should probably move to spark.