I am training a deep learning model for traffic prediction. When I use 10 months of data for training (validation split: 10%) and 2 months for testing. The loss curve looks like this:
.
and the performance of model is very bad (rmse: 41.98). But when I reduce the training data and use 9 months of data (validation split: 10%) and increase the test size (3 months), the performance is excellent (rmse: 7.92). The loss curve looks like this:
In the first training, the training data is large and test data is small. Counterintuitively, it performed poor compared to second training where training data is small and test size is higher.
Can somebody point out what is happening is these two trainings? Also, what these two learning curves tells about the learning? Also, in second training, there is a sudden spike after 20 epochs and steep decline between 120 and 140 epochs. What is happening in these points?
Other parameters:
- learning rate: 1e-4
- early stop patience: 30
- Optimizer: Adam