Problem type: Multi class classification
The dataset contains around 1,000 samples (simulated dataset of sensor signals), where each sample is 2D i.e (1000 * 1000 * 8). Additionally, I have a few real world data, which is of shape (100 * 1000 * 8)
I split my data into training and validation from the simulated data set and use the real world data as test set.
I performed cross validation with 5 folds + data augmentation techniques since I have low samples on train set, which also takes care of data imbalance.
I built a neural network architecture (CNN), but evaluated my model using accuracy metrics.
The validation data performs really well around 85% accuracy, where as my test data outputs only 60% accuracy.
What does this mean? Why is my model not performing well on the test data set (real world data set)?
I found a similar question asked before, but I am not able to understand what exactly the accepted answer is explaining. So I am posting this question, once again, to gain more insight into the problem and required actions.