I have a Time Series Forecasting problem. You can think of it as predicting the daily closing prices of Apple stocks. My data is divided into 4-day segments, and the forecasting is based on predicting the 5th day. To fine-tune the model for this problem, I'm using Sklearn's TimeSeriesSplit method as follows:

tscv = TimeSeriesSplit(n_splits=15, test_size=5)

The reason for setting test_size to 5 is to use the last 4 days as X_test and try to predict the 5th day as y_test. Setting n_splits to 15 allows me to perform this backward for the last 15 days, ensuring that I'm testing on recent prices.

My first question is: "Could there be a logical flaw in what I'm doing here? Am I using Cross-Validation incorrectly? Because the mean and standard deviation of my errors are constantly changing."

My second question is: "In this problem, as you may know, new data is added every day, and the model is trained and tested on updated data. Since I measure the error as Mean Squared Error, the importance of each error is high. However, since the test data changes daily, the values I measure one day do not hold for the next day. Using a fixed dataset is not suitable as financial data is subject to change. I don't believe that a model trained on data from two weeks ago can accurately generalize to today. How can I measure stability as closely as possible in this scenario?"

Thank you in advance for your insights. I haven't included code examples as my question is theoretical.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.