1
$\begingroup$

I have a binary NLP classification task to identify text that talks about a target topic from millions of sentences. Between 5-10% of sentences are positive, the rest is negative.

I have trained several models on about 1,000 manual annotations on a random sample, of which 30% positive and 70% negative, in which I over-sampled positive cases after seeing that the precision on positive cases was too low. My initial exploration showed that negative cases are easier to identify, so I included more positives.

Using cross validation or training/validation splits, I get an accuracy between 80% and 90%, which would be sufficient for the project.

However, when I run the models on the actual data and observe a sample of results, the results are way poorer (about 60% accuracy). This occurs in a very similar way with all models I'm using (random forests, neural nets, etc). Is it a problem of overfitting or the training sample not being representative? How can I diagnose and solve the problem?

Improve this question
$\endgroup$
5
  • $\begingroup$ how did you choose the 1000 samples from the dataset? $\endgroup$
    – spectre
    Commented Oct 29, 2021 at 11:41
  • $\begingroup$ Does the overall "actual" data also have 30-70 class ratio? $\endgroup$
    – user16584277
    Commented Oct 29, 2021 at 13:12
  • $\begingroup$ I edited the text $\endgroup$
    – Strabonio
    Commented Oct 29, 2021 at 14:07
  • $\begingroup$ I suggest to downsample the negative class, that is create balanced dataset including only as many negatives as positives $\endgroup$
    – Nikos M.
    Commented Oct 29, 2021 at 15:09
  • 1
    $\begingroup$ When you did cross-validation, did you test on a resampled data? If so this a mistake: only the training set should be resampled. If the test set doesn't follow the true distribution, then the performance is biased. Also you should use precision/recall/f1-score (or observe the confusion matrix) as performance measure, this way you could analyze what happens. $\endgroup$
    – Erwan
    Commented Oct 30, 2021 at 23:38

1 Answer 1

Reset to default
1
$\begingroup$

There is often a performance drop from training data to unseen data with text data. This is because of the out-of-training-sample data. There are often tokens that do not appear during model training but appear in unseen data.

One option is to make sure as much of the total vocabulary is represented in the training data.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.