I am trying to build a Decision-Tree model on top of a dataset that is about 10G in size on my local computer. However, I only have 8G memory. What I am doing now is just random sampling certain subset of the data, trying different model parameters and checking the prediction performance with the scikit-learn package. Still I need a final model based on the model parameters from those random sampling results, but obviously I cannot build a model on top of the full dataset. What can I do with this problem? I am a newbie of machine learning, any suggestions are welcome. Thanks!
Whether you're using this for a hobby or for your job, I would recommend EMR on Amazon Web Services. The name is anachronistic (it's no longer just for map reduce), but EMR enables you to quickly stand up an Apache Spark cluster of several machines. Spark comes with machine learning libraries for building trees, and can scale to however many machines you need (to the point that you no longer face memory constraints). EMR is cheap because you're charged hourly and don't have to buy your own hardware. Apache Spark is great for your resume, and it's also a practical skill because it's generally the first tool most data scientists turn to when working with data sets that are too large for a single machine.
Since you're using Python and scikit-learn you could have a look at one of the following online learning algorithms:
All of these online learning algorithms (in particular with SGD) allow for streaming the data through memory one entry at a time. Your memory would be more than sufficient for this approach. In scikit-learn this is implemented via the partial_fit() method. More on this out-of-core approach can be found in the scikit-learn user guide.
If you want stick to tree based algorithms, you can have a look at the xgboost package which also allows for streaming data through memory. However, this approach is a little more involved because it only accepts data in the LIBSVM format in order to parse it in the memory cache preserved for xgboost. Also, it doesn't allow for parameter tuning, since xgboost works on numpy objects and converting from LIBSVM to numpy dumps the data from the cache to the main memory and therefore doesn't scale.
You could also use the Databricks Community Edition which let's you spin up Spark clusters (of limitied size for the free edition) where you can run pyspark or plain python scripts.
In general, more training examples means improvement in learning but you can also get a very good (and nearby to the optimal score) if you just fit a good algorithm on a subset of your data set that has enough training examples. Here are a few things you can do in your current case :
Take a subset of the data say about 4-5GB. The only thing you need to consider is that your target labels should be nearly stratified in the subset other wise the model will perform poorly.
Apply PCA on your data and try to minimize the number of features. There may be features that are just redundant in your subset of data.
Apply the algorithm, best suited for your dataset, on your current subset. You may need to do a bit of research for that.
And above all, I suggest one thing. Training such a large dataset on a local machine can be too much of pain. So, it's better if you deploy your model on AWS or Google's Machine Learning platform provided by them on cloud.
I hope it helps!!
You can also use some out of core libraries like GraphLab (watch out though, GraphLab is free only for educational purposes). GraphLab works the same way as Scikit-learn but can run 'out of core' (meaning it isn't limited by the amount of memory you have).