Machine Learning Best Practices for Big Dataset

Question

I am about to graduate from my Master and had learnt about machine learning as well as performed research projects with it. I wonder about the best practices in the industry when performing machine learning tasks with Big Datasets (like 100s GB or TB). Appreciate if fellow data scientists can share their experience. Here are my questions:

1. Obviously, very large datasets take longer time to train (can be days or weeks). Many times we need to train various models (SVM, Neural Network, etc.) to compare and find better performance model. I suspect, in industry projects, we want the results as quick as possible but produce the best performance. Are there any tips for reducing the training & testing time? If you recommend subsetting the dataset, I will be interested to learn how best to subset the dataset to cover all or majority of scenarios from the dataset.
2. We know that performing cross validation is better as it may reduce over-fitting. However, cross validation also takes time to train and the model trained with cross validation may not be implemented straight (speaking from python sklearn experience: I need to train the model with dataset again after cross validation testing for it to be implemented). Do you normally do cross validation in your big data projects or getting by with the train-test split?

Appreciate the feedback.

Answer 1

I'll list some practices I've found useful, hope this helps:

1. Irrespective of whether the data is huge or not, cross validation is a must when building any model. If this takes more time than an end consumer is willing to wait, you may need to reset their expectations, or get faster hardware/software to build the model; but do not skip cross validation. Plotting learning curves and cross-validation are effective steps to help guide us so we recognize and correct mistakes earlier in the process. I've experienced instances when a simple train-test set does not reveal any problems until I run cross-fold validations and find a large variance in the performance of the algorithm on different folds.

2. Before sizing up a dataset, eliminate the records with missing values of key variables and outliers, columns of highly correlated variables, and near zero variance variables. This will give you a much better estimate of the real usable dataset. Sometimes you may end up with only a fraction of the available dataset that can actually be used to build a model.

3. When sizing up a dataset for building a model, it is easier to estimate the computing resources if you enumerate the dataset in rows and columns and memory size of the final numeric matrix. Since every machine learning algorithm is ultimately going to convert the dataset into a numeric matrix, enumerating the dataset size in terms of GBs/TBs of raw input data (which may be mostly strings/textual nominal variables/etc.) is often misleading and the dataset may appear to be more daunting and gigantic to work with than it is.

4. Once you know (or estimate) the final usable size of your dataset, check if you have a suitable machine to be able to load that into memory and train the model. If your dataset size is smaller than memory available/usable by the software, then you need not worry about the size any longer.

5. If the dataset size is larger than the memory available to train a model, then you could try these approaches (starting from the simplest ones first):

   • Use a machine with more memory: If you're using a cloud service provider then the simplest approach could be just to provision more memory and continue building the model as usual. For physical machines, try to procure additional RAM, its price continues to reduce and if your dataset is going to remain this big or grow bigger over time, then it is a good investment.
   • Add nodes to the cluster: For Hadoop and Spark based cluster computing deployments, training on a larger data-set is as easy as adding more machines to the cluster.
   • Quite often classification tasks require training on data with highly imbalanced classes, the ratio of positive to negative classes could sometimes be as large as 1:1000 or more. A straightforward method to improve accuracy in these cases is to either over-sample the minority class or under-sample the majority class, or do both together. If you have a large dataset, under-sampling the majority class is a very good option which will improve your algorithm's accuracy as well as reduce training time.
   • Build an ensemble: Split the dataset randomly and train several base learners on each part, then combine these to get the final prediction. This would most effectively make use of the large dataset and produce a more accurate model. But you need to spend more time to carefully build the ensemble and keep clear of the usual pitfalls of ensemble building.
   • If you're using an ensemble, train many single-thread models in parallel. Almost all ML software provide features to train multiple models on different cores or separate nodes altogether.
   • Evaluate multiple different algorithms on the time taken to train them for your specific dataset vs. their accuracy. While there is no universal answer, but I've found when using noisy data, SVMs take much longer time to train than carefully built ensemble of regularized regression models, but may be only slightly more accurate in performance; and a well built neural network may take a very long time to train as compared to a CART tree, but perform significantly more accurately that the tree.
   • To reduce time taken to build the model, try to automate as much of the process as you can. A few hours spent automating a complex error-prone manual task may save your team a hundred hours later in the project.
   • If available, use those algorithm implementations which use parallel processing, sparse matrices and cache aware computing, these reduce processing time significantly. For example, use xgboost instead of a single-core implementation of GBM.
   • If nothing else works, train the model on a smaller dataset; as Emre has suggested in his answer, use learning curves to fix the smallest sample size required for training the model, adding more training records than this size does not improve model accuracy noticeably. Here is a good article which explores this situation - largetrain.pdf

Answer 2

The question is, how much data does it take to saturate your model? To determine this you can plot learning curves with varying amounts of data, perhaps scaling up/down in size by a constant factor. If training on all the data is not feasible, the learning curve can help you make an informed trade-off.

Model saturation also comes into play with cross-validation. If you are not near saturation, using a small number of folds will give you pessimistic performance figures because you will be training your model on less data than it can make use of.

Finally and conversely, you can use a more complex model instead of "resizing" the data to suit the model.

Welcome to DataScience.SE.