I am using 3-fold cross validation and a grid search of the C and gamma parameters for a SVC using the RBF kernel I have achieved a classification score of 84%.
When testing against live data the accuracy rate is 70% (1500 samples used). However, when testing against an un-seen hold out set the accuracy is 86% (8800 samples, 20% of the original dataset).
The training and holdout data set have even distribution of the 3 classes.
What could be the cause of this large discrepancy? It does not seem to be over fitting judging by the performance of the model with the hold out set?
EDIT:
How did you split the data set? The data was originally in sequential order. I wrote a script to randomly split each sample between the train and hold out set, making use of a CSPRNG. Then at the end a report was automatically generated to display the distribution of each class in each set. The distribution very near equal.
How did you do the grid search?
Through the SKlearn SVC grid search method (GridSearchCV
).
Is there any overlap between the data points used during grid search and the un-seen hold out set? No overlap, they are all from unique time stamps in the initial set.
Does the live data come from the same distribution as the other? Yes the live data comes from the same source and the distribution is roughly the same.
How do you know? I have a script to count up the occurrences of each class in the data set.