5
$\begingroup$

I am using 3-fold cross validation and a grid search of the C and gamma parameters for a SVC using the RBF kernel I have achieved a classification score of 84%.

When testing against live data the accuracy rate is 70% (1500 samples used). However, when testing against an un-seen hold out set the accuracy is 86% (8800 samples, 20% of the original dataset).

The training and holdout data set have even distribution of the 3 classes.

What could be the cause of this large discrepancy? It does not seem to be over fitting judging by the performance of the model with the hold out set?

EDIT:

How did you split the data set? The data was originally in sequential order. I wrote a script to randomly split each sample between the train and hold out set, making use of a CSPRNG. Then at the end a report was automatically generated to display the distribution of each class in each set. The distribution very near equal.

How did you do the grid search? Through the SKlearn SVC grid search method (GridSearchCV).

Is there any overlap between the data points used during grid search and the un-seen hold out set? No overlap, they are all from unique time stamps in the initial set.

Does the live data come from the same distribution as the other? Yes the live data comes from the same source and the distribution is roughly the same.

How do you know? I have a script to count up the occurrences of each class in the data set.

Improve this question
$\endgroup$
5
  • $\begingroup$ How did you split the data set? How did you do the grid search? Is there any overlap between the data points used during grid search and the un-seen hold out set? Does the live data come from the same distribution as the other? How do you know? Can you add to the question to clarify these points? $\endgroup$
    – D.W.
    Apr 4, 2018 at 5:21
  • $\begingroup$ @D.W. thanks for taking the time to get back to me. Please see my updated question for your answers $\endgroup$
    – sousdev
    Apr 5, 2018 at 15:16
  • $\begingroup$ You might check that cross-validation was done only on the training set (so cross-validation in turn splits up the training set). $\endgroup$
    – D.W.
    Apr 5, 2018 at 16:15
  • $\begingroup$ @D.W. thanks, I can confirm it was only performed on the training set $\endgroup$
    – sousdev
    Apr 6, 2018 at 19:33
  • $\begingroup$ To make sure that you avoided overfitting, randomize the labels. If your algorithms makes good predictions for randomized labels, it is a clear indicator that you are overfitting. If, on the other hand, your performance goes down for random labels, then you have most likely not overfitted the model and the reason for the descrepancy probably lies somewhere else. $\endgroup$
    – Eulenfuchswiesel
    Jun 5, 2018 at 7:21

2 Answers 2

Reset to default
1
$\begingroup$

It seems likely that the live data is different somehow from your other data. Cross-validation shows a 84% accuracy, and accuracy on the held-out set is 86%, which is pretty consistent and does not indicate overfitting. Accuracy on the live data is 70%, which is significantly different. That suggests that live data is somehow different in ways that are important to the classifier. Perhaps concept drift has occurred.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I was also thinking the new live data may be different in some way, do you know of any way to test this theory? $\endgroup$
    – sousdev
    Apr 6, 2018 at 19:39
  • $\begingroup$ @sousdev, You have to look at how the data was acquired and whether there might be differences in the process used to generate the training data vs the process used to generate the live data. That difference might be as simple as "passage of time", if that could somehow affect the process or distribution of the data. You can read more about concept drift to learn more about the subject and ways of dealing with it -- spend some time reading about that here before asking here, and if you have a specific question about that topic, post a new question using the 'Ask Question' button. $\endgroup$
    – D.W.
    Apr 6, 2018 at 19:40
0
$\begingroup$

It happens majority of time that accuracy on train data is different than accuracy on test data. This may happen because test data is quite different than test data. Model is not able to perform well on unseen data(i.e. test data).

In order to minimize the discrepancy one need to make sure that model while training consider all possibilities so that it consider the majority of the variance.

Many a times while trying to learn variance, accuracy of the model get affected because of over fitting or under fitting scenario.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.