Suppose I have a set of already labeled documents -- some of them are job descriptions/postings (these are documents of interest), and some of them are not. I wonder what kind of method would allow me to build a model that can generalize to new data, specifically new job descriptions that may be very different from the already labeled job descriptions.
What I have done so far (in Python):
(1) Trial 1: I tried the classic bag-of-words + TfidfVectorizer + binary classification (LogisticRegression specifcally) approach. With parameter tuning based on the train + valid sets, precision and recall on the test set could reach over 98%. However, after I built the model, I collected a new set of documents (again, some are job postings and others are not), and asked annotators to label these new data. I then used the model to classify these new docs. Perhaps expectedly, the precision and recall dropped pretty drastically to, ~65% and ~40%.
(2) Trial 2: I thought perhaps using tfidf alone might cause some overfitting, so I tried applying TruncatedSVD (with 100 components) on the TfidfVectorizer (bag-of-words + TfidfVectorizer + TruncatedSVD + binary classification). Again, I tuned the parameters using the train and valid sets, and the precision and recall were similarly high on the test set. When I applied this new model to the newly collected data, the recall improved a bit, from 40% to 50%, but the precision dropped a bit, to 58%.
So my question is: is there a way to leverage the initial set of coded documents to build a kind of model that can generalize better to new data set. My specific interest is identifying job descriptions/postings. It doesn't have to be a binary classification task since the non-job-descriptions can be much more varied than job descriptions/postings
