I have a binary NLP classification task to identify text that talks about a target topic. I train several models on about 1,000 manual annotations on a random sample, of which 30% positive and 70% negative. The full dataset has a ratio closer to 10/90 positive-negative, so I over-sampled positive cases after seeing that the precision on positive cases was too low, assuming that negative cases are easier to identify.

Using cross validation or training/validation splits, I get an accuracy between 80% and 90%, which would be sufficient for the project.

However, when I run the models on the actual data and observe a sample of results, the results are way poorer (about 60% accuracy). This occurs in a very similar way with all models I'm using (random forests, neural nets, etc). Is it a problem of overfitting or the training sample not being representative? How can I diagnose and solve the problem?