Time series analysis model evaluation performance metrics integration in time series application

Question

After study in time series analysis, I recognized RMSE and MAPE are the best evaluation metrics for used model in real time series application. But my queries are below as this is my first practice application project in time series analysis:

If I use past 1 year data for training and forecast for next 30 days but we will get actual data after 30days in real time. How could we integrate these RMSE and MAPE with time-series application?

As I heard from some data scientist if I skip first training cycle and calculate these RMSE and MAPE in next retraining cycle for past training cycle where I have actual data and forecast data, Is this approach useful ?

I concerned about calculated RMSE and MAPE could help to improve model performance in real time.

Could the evaluation metrics calculation every time on retaining useful from production point of view?

My thanks in advance

Answer 1

I am assuming you have a model running in production and it retrains periodically (example: every month) and it forecasts for the next X days (example: 30 days) and you are trying to evaluate the model using RMSE and MAPE (If this assumption is wrong, please clarify it in your question)

If I use past 1 year data for training and forecast for next 30 days but we will get actual data after 30days in real time. How could we integrate these RMSE and MAPE with time-series application?

you will have to back test your model, i.e. predict with your model on a day in the past from which you will have 30 forecast-ed values and 30 actual values. Then the aggregation happens in mean part of the Root Mean Square Error or Mean Average Percentage Error. Ideally, your model should not have seen the evaluation data during the training phase.

As I heard from some data scientist if I skip first training cycle and calculate these RMSE and MAPE in next retraining cycle for past training cycle where I have actual data and forecast data, Is this approach useful ?

Yes, this is one of the ways to evaluate the model.

Could the evaluation metrics calculation every time on retraining useful from production point of view?

It is a good practice (unless it costs too much) to monitor the model in production regularly, for this you will have to estimate the evaluation metric in the frequency that is most suitable for you.