I have a Time Series Forecasting problem. You can think of it as predicting the daily closing prices of Apple stocks. My data is divided into 4-day segments, and the forecasting is based on predicting the 5th day. To fine-tune the model for this problem, I'm using Sklearn's TimeSeriesSplit method as follows:
tscv = TimeSeriesSplit(n_splits=15, test_size=5)
The reason for setting test_size to 5 is to use the last 4 days as X_test and try to predict the 5th day as y_test. Setting n_splits to 15 allows me to perform this backward for the last 15 days, ensuring that I'm testing on recent prices.
My first question is: "Could there be a logical flaw in what I'm doing here? Am I using Cross-Validation incorrectly? Because the mean and standard deviation of my errors are constantly changing."
My second question is: "In this problem, as you may know, new data is added every day, and the model is trained and tested on updated data. Since I measure the error as Mean Squared Error, the importance of each error is high. However, since the test data changes daily, the values I measure one day do not hold for the next day. Using a fixed dataset is not suitable as financial data is subject to change. I don't believe that a model trained on data from two weeks ago can accurately generalize to today. How can I measure stability as closely as possible in this scenario?"
Thank you in advance for your insights. I haven't included code examples as my question is theoretical.
