4
$\begingroup$

I was wondering if anyone has any advice on where to start digging for this problem. I have a model which has gone through development and all train/cv/test data sets now perform above 95% both for accuracy and F-Score. The total development data set is around 60k samples, with a 2/3 split for positive and negative samples. These samples are based on extracts for the months of Jan to Nov of last year. Final test results were:

Precision: 0.9751 Recall: 0.9320 Accuracy 0.9693 F score 0.9531

However, the first runs in production showed a very high precision:95%+ but a very low recall:~50%. Accuracy = 48%, FScore = 68%.

Any thoughts from the group on this, where to look, potential causes. We will run this over the next couple of months, as we may have exceptional variations due to the Xmas period, but we were surprised. Any help appreciated. Thanks

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Could be way too many things... overfitting? biased sample of training data? wrong train/test/CV process? you didn't measure recall on your test set, nor F-score on your production data, but are trying to compare them. $\endgroup$
    – Sean Owen
    Jan 9, 2019 at 16:09
  • $\begingroup$ I have made some changes, hopefully this might give some ideas. During test we had a lot of problems with overfitting, which we finally managed with a combination of dropout, and feature enhancement. With a recall problem we normally look to features, and in which case this maybe consistent with the past. But looking to any other suggestions as well $\endgroup$
    – 1961DarthVader
    Jan 9, 2019 at 16:40
  • $\begingroup$ The most likely explanation is that you have over-fit, and real data from December doesn't really follow the distribution of data from Jan - Nov. $\endgroup$
    – Sean Owen
    Jan 9, 2019 at 22:03
  • $\begingroup$ December is a strange month, so while we debug further, we will let the application run thru Jan. Thanks $\endgroup$
    – 1961DarthVader
    Jan 10, 2019 at 6:25

1 Answer 1

Reset to default
2
$\begingroup$

This happened to me in the past.

The first question I would ask myself will be, how am I splitting train, validation and test sets? The second one would be, are my production data in the same domain as of those in train/validation/test?

Sometimes a simple split of data might still result in testing data leaking into the validation and even test data, this is the case of domains where we want to study the behaviour of users, normally you will not want the same users in train/test/validation sets (that is, ideally a user should remain within the same data set).

Apart from that, looks like you might just be overfitting the data.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Given the trouble we had with overfitting during development, I was hoping that we would not have to go there again, but it it seems a sensible place to start. Also we know December is a strange month, so there could be exceptional data there. While we run other tests, we will let the solution run thru Jan and see how we perform. Thanks $\endgroup$
    – 1961DarthVader
    Jan 10, 2019 at 6:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.