0
$\begingroup$

I have build a model using transactions data trying to predict the value of future transactions. The main algorithm is Gradient Boosting Machine. The overall accuracy on the testset is fine and there is no sign of overfitting. However, a small change in the training set creates radical change in the model, and in the predictions. But even when the testset change a little the overall accuracy is stable.

The time period is from 2005 to today and when a single day is added to the dataset predictions change drastically (e.g. +/- 10%). If multiple training are perform on the same training set, the predictions are the same.

I have test Light GBM(2.1.0) and XGBoost(0.60) with Python 3.6 on Windows 10. A seed is set and I train the model on CPUs. I have tried to increase the number of iterations to a high number and adding a specific seed to the bagging parameters.

This blogpost discuss brefly that issu without giving any solutions.

$\endgroup$
  • $\begingroup$ May be that your test set does not adequately represent the "average" of the data (e.g. some special cases in there which are not in the training set). Did you try cross validation on different test sets? What is the number of observations in test/train $\endgroup$ – Peter May 27 '19 at 15:50
  • $\begingroup$ The trainset is about 1.3M obs and the testset 200k. But the symptoms of the problem are not seen in the cross-validation. Everyday, the model predict new values for the transaction price of good and those prices changes a lot. $\endgroup$ – Charles_de_Montigny May 27 '19 at 18:05
  • $\begingroup$ Okay, so there is an important time dimension. When there is a lot random walk "white noise" it is hard to make predictions. Not sure what model you actually use, but could it be, that the time dimension is underexposed? I worked with NN LSTM on stock market prices (similar problem I guess), and it turned out quite well. $\endgroup$ – Peter May 27 '19 at 18:10
  • $\begingroup$ How are you validating the model? You state that the predictions change a lot with the addition of one day which implies a time series element. Are you using a validation scheme that accounts for this crucical fact if this is true? The predictions might change a lot because of the time element, if you are using time as a variable, for example. If the validation results are satisfactory even with rapid changes in predictions why does this matter? $\endgroup$ – aranglol May 28 '19 at 13:36
  • $\begingroup$ For time series validation, you should make sure that your hold out set is chronologically after the train set. You can't do cross-validation (barring some special forms of rolling origin CV) with time series data. $\endgroup$ – Victor Ng Oct 25 '19 at 12:36
0
$\begingroup$

A good way to avoid this problem is to add noise to your training dataset this will made your model more robust and less versatile.

there is a different way of adding noise, often a gaussian noise is added it depends to the kind of data you have.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.