2
$\begingroup$

I created a model to solve a time series forecasting problem. I had a limited amount of time series with which I could train the model therefore I decided to augment the data. The data augmentation strategy I used is quite basic but has shown to increase the accuracy of my model.

I wrote my own data_generator which I use to train my model with, using the fit_generator function in keras. Essentially it takes in the whole training data set that I have, shuffles all the time series and the augmentation process takes place specifically in each batch. In each batch I randomly pick, per time series in the batch, start and end points, so that each batch contains varying length slices of each series within the batch. This creates obviously an almost endless stream of data but it is entirely reliant on the number of epochs the model is run for as the dataset is not augmented upfront. No noise or anything is applied to the data set, the augmentation is purely from varying the lengths of the time series and the start and end points of the series.

I observe that my loss continues to decrease over time I have tried 100, 500, 1000, 5000 and 10,000 epochs. In general the accuracy of the model predictions does get better but at some point with diminishing returns. It is hard to say when as I am still tuning the model architecture and hyperparameters.

Does such an augmentation strategy affect how I can interpret the loss of model? As the longer I train the model for the more "new" data it sees instead of constantly seeing the same data and training on it.

$\endgroup$
3
$\begingroup$

If you can be sure that the model is not seeing the same instances repeatedly then there is very good chances that your model is not overfitting and that is precisely what you can measure from your validation set, you should see continuously downward loss which will eventually plateau at the local optimum that has been eventually reached by your model, that is the best possible solution attained given the starting conditions. If your model would be overfitting then you should see the loss function start increasing for the validation.

The best way to know if you are overfitting is to take your model and then apply it to a completely new dataset and measure the performance. If the performance is good, then you are fine.

p.s. To overcome falling inside a bad optimum you can train multiple models each with different starting conditions for the model parameters and then make an ensemble classifier.

$\endgroup$
1
$\begingroup$

I have a somewhat similar model, where I limit the number of epochs for training, simply from the efficiency considerations. I let the model train for about 40 epochs, while I include the drop-out feature in the training. In my case, it is a three layers fully connected net.

Using validation set I save the model from the epoch which improves the validation loss, but let it finish full 40 epochs, since I believe sometimes it can converge into local minima and then find better optimum since there is a drop-out.

In your case, I think that the lengths variations also introduce randomization and prevent overfitting (similar to drop-out). But I think you have to monitor the validation loss because it is possible your model will converge to a point with a better training loss, but the validation loss can be less than optimal...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.