I am new to ML and have been reading online about training bottlenecks when there are frequent updates to data.

Let's say I have a built a model based on a dataset of 10M records.

Now, in another 2 months, I might receive another 1M records which we would like to feed into our model as well.

Similarly this goes on for every 2 months. We would like to update/train our model with latest data as and when it's available

1) But for example, let's say the training takes 1 week for every new data update

2) Any suggestions on how we can minimize the training time (when we train every 2 months?)?

3) Should we select a representative sample from 1 Million datapoints? Is that good enough

4) I understand it's all about tradeoff but I am curious to know whether I am missing any known approaches to save training time? I am thinking representative sample can reduce the sample size and help us fasten the training process

Can you share your suggestions on this?

  • 1
    $\begingroup$ Are you saving the trained model? $\endgroup$ – Harshith Feb 14 at 4:59
  • $\begingroup$ Yes, it will be saved $\endgroup$ – The Great Feb 14 at 5:12
  • $\begingroup$ You take the saved model and retrain on new 1M records. Now you want to reduce the training time? Is this right? $\endgroup$ – Harshith Feb 14 at 5:20
  • $\begingroup$ Yes, you are absolutely right $\endgroup$ – The Great Feb 14 at 5:35

From what I understand - your problem is "sample selection bias" problem. Any kind of pattern to select a subset out of large data may lead to bias. This raises two question.

  1. How to choose? Random/stratified random (If you have multiple classes) under sampling to obtain a smaller subset.
  2. How big to choose? we can set percentage of undersampling.

Reducing training time : We can perform preprocessing : dimensionality reduction techniques to remove correlated data by aplying techniques like- PCA. Or Sparse reconstruction methods - Transform data to sparse data and then process.

| improve this answer | |
  • $\begingroup$ When the model is already trained based on a dataset of say 10M records with certain features, now I need to make sure that the new data that I receive every 2 months fits the same format (has same features etc) as the data that was used to train earlier. So now my qustion is how can we make this efficient? If I do PCA, then I will lose the feature info which old training data had. Does my explanantion help you understand the problem? $\endgroup$ – The Great Feb 14 at 3:42
  • $\begingroup$ upvoted for the response $\endgroup$ – The Great Feb 14 at 3:42
  • 1
    $\begingroup$ Yes, I understand. You already have a trained model. You dont want to train from scratch. Check this out if it helps. [link] (stats.stackexchange.com/questions/352750/…) $\endgroup$ – mrin9san Feb 15 at 6:25
  • $\begingroup$ If you do PCA like techniques, you reduce data redundacy related to one particular feature. Redundancy in different components in different data may vary. $\endgroup$ – mrin9san Feb 15 at 6:35
  • $\begingroup$ So, PCA can't be applied. Am I right to understand this way? $\endgroup$ – The Great Feb 16 at 2:00

You have a big dataset and you get new instances//data every 2 months.

First you should select with which data you want to train. Since your data is big and there is the probability than data from 2 years ago is not as relevant as the data from the last month you can consider doing a Roll out// slidding window validation. This way you will only select the latest data and your model will expend less time trainning.

Rolling Window

In this case (from a previous question from here) you can see how you don't need to train with all the data but with the most recent part.

This approach can work for you and can offer a boost in your trainning time, a boost in your model since it is only considering the most recent time and a proposed sampling method.

| improve this answer | |
  • $\begingroup$ No, I don't get new features every 2 months. I mean data gets updated regularly every 2 months. Consider it as transaction system where you have new transactions coming in every day. So, we would like to train our model regularly (in addition to old training) and expose it to all variations in data. So my question now is how can I make this training efficient? $\endgroup$ – The Great Feb 14 at 3:39
  • $\begingroup$ Thanks for the response. Upvoted $\endgroup$ – The Great Feb 14 at 3:40
  • $\begingroup$ Sorry I meant new data. $\endgroup$ – Carlos Mougan Feb 14 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.