I am building a supervised machine learning model to generate forecast.

So I would have historic data like this:

SKU, Month, .... other features, Actual Volume

That I can use a model to generate forecast, using the actual volume as the label.

Of course, there would be a variance between the forecast volume and the actual volume

What are the proper ways to leverage such data, without generating any data leakage, to incorporate such info to train the model to minimize the variance?

Should the data be fed back to the data with moving average, etc. and retrain, or is there other better strategy to properly assess the performance of the model and learn from it?

The data will be time series data with various features such as exchange rate, salesperson, etc.

  • $\begingroup$ I may misunderstand, but are you saying that you want to include the prior month's forecast error into the prediction for the current month's volume? This seems like a bad idea, as the forecast residuals should be random noise. If there is information in the residuals, then you should be taking that information into account in your original model. $\endgroup$ Jan 9 '19 at 18:04
  • $\begingroup$ Thank you for your insight. I know ideally the original model will take care of the residual, but I am not 100% convinced that the residuals will always be random noise. That's why I want to have a process in place to track the performance of the model over time to hopefully discover ways to continuously improve the model. Also, I want to observe the bias of the prediction, and sometimes, some systems would provide their own ML algorithm and predictions. We will build our model, but we might also include predictions from other systems and hopefully those will help improve the algorithm too. $\endgroup$
    – The Lyrist
    Jan 10 '19 at 16:40

I'm glad to see this question because this site gets such few questions on models that are actually in a production state.

If I was in your position, I would start to think about how I'm going to use this algorithm on a go-forward basis and start to log everything. Every new prediction that your algorithm makes is also a new data point for training.

So you need to take the time to create a feedback loop and (1) take new datapoints, put them into training, (2) re-train your algorithm, (3) deploy a new version and then (4) use that revised algorithm. You repeat this entire process on a continual basis throughout the life of your project. From the technical side, it's an exercise in re-inforcement learning since your algorithm won't start from a base of zero knowledge. Also, most of the steps I described will most likely require you to write new code, most of which will be related to process and workflow, not necessarily data science or algorithmic.

Finally, as an FYI, it is said that there are multiple algorithms out there that carry out these types of cycles on a continual basis, literally updating on a minute-by-minute (or faster!) basis, always providing answers on the latest available data points.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.