How to train my model efficiently?

Question

I am new to ML and have been reading online about training bottlenecks when there are frequent updates to data.

Let's say I have a built a model based on a dataset of 10M records.

Now, in another 2 months, I might receive another 1M records which we would like to feed into our model as well.

Similarly this goes on for every 2 months. We would like to update/train our model with latest data as and when it's available

1) But for example, let's say the training takes 1 week for every new data update

2) Any suggestions on how we can minimize the training time (when we train every 2 months?)?

3) Should we select a representative sample from 1 Million datapoints? Is that good enough

4) I understand it's all about tradeoff but I am curious to know whether I am missing any known approaches to save training time? I am thinking representative sample can reduce the sample size and help us fasten the training process

Can you share your suggestions on this?

Answer 1

From what I understand - your problem is "sample selection bias" problem. Any kind of pattern to select a subset out of large data may lead to bias. This raises two question.

1. How to choose? Random/stratified random (If you have multiple classes) under sampling to obtain a smaller subset.
2. How big to choose? we can set percentage of undersampling.

Reducing training time : We can perform preprocessing : dimensionality reduction techniques to remove correlated data by aplying techniques like- PCA. Or Sparse reconstruction methods - Transform data to sparse data and then process.

Answer 2

You have a big dataset and you get new instances//data every 2 months.

First you should select with which data you want to train. Since your data is big and there is the probability than data from 2 years ago is not as relevant as the data from the last month you can consider doing a Roll out// slidding window validation. This way you will only select the latest data and your model will expend less time trainning.

In this case (from a previous question from here) you can see how you don't need to train with all the data but with the most recent part.

This approach can work for you and can offer a boost in your trainning time, a boost in your model since it is only considering the most recent time and a proposed sampling method.