I am about to graduate from my Master and had learnt about machine learning as well as performed research projects with it. I wonder about the best practices in the industry when performing machine learning tasks with Big Datasets (like 100s GB or TB). Appreciate if fellow data scientists can share their experience. Here are my questions:
- Obviously, very large datasets take longer time to train (can be days or weeks). Many times we need to train various models (SVM, Neural Network, etc.) to compare and find better performance model. I suspect, in industry projects, we want the results as quick as possible but produce the best performance. Are there any tips for reducing the training & testing time? If you recommend subsetting the dataset, I will be interested to learn how best to subset the dataset to cover all or majority of scenarios from the dataset.
- We know that performing cross validation is better as it may reduce over-fitting. However, cross validation also takes time to train and the model trained with cross validation may not be implemented straight (speaking from python sklearn experience: I need to train the model with dataset again after cross validation testing for it to be implemented). Do you normally do cross validation in your big data projects or getting by with the train-test split?
Appreciate the feedback.