0
$\begingroup$

Suppose I have a set of already labeled documents -- some of them are job descriptions/postings (these are documents of interest), and some of them are not. I wonder what kind of method would allow me to build a model that can generalize to new data, specifically new job descriptions that may be very different from the already labeled job descriptions.

What I have done so far (in Python):

(1) Trial 1: I tried the classic bag-of-words + TfidfVectorizer + binary classification (LogisticRegression specifcally) approach. With parameter tuning based on the train + valid sets, precision and recall on the test set could reach over 98%. However, after I built the model, I collected a new set of documents (again, some are job postings and others are not), and asked annotators to label these new data. I then used the model to classify these new docs. Perhaps expectedly, the precision and recall dropped pretty drastically to, ~65% and ~40%.

(2) Trial 2: I thought perhaps using tfidf alone might cause some overfitting, so I tried applying TruncatedSVD (with 100 components) on the TfidfVectorizer (bag-of-words + TfidfVectorizer + TruncatedSVD + binary classification). Again, I tuned the parameters using the train and valid sets, and the precision and recall were similarly high on the test set. When I applied this new model to the newly collected data, the recall improved a bit, from 40% to 50%, but the precision dropped a bit, to 58%.

So my question is: is there a way to leverage the initial set of coded documents to build a kind of model that can generalize better to new data set. My specific interest is identifying job descriptions/postings. It doesn't have to be a binary classification task since the non-job-descriptions can be much more varied than job descriptions/postings

$\endgroup$
0
$\begingroup$

You should at least try pre-trained embedding vectors. TfidVectorizer is particularly sensitive to out-of-vocabulary words, which are likely to appear if you're trying transfer learning to a new domain. The GloVe embeddings [1] have a dictionary of 400k vectors, unless you're working with documents from a technical domain they should provide some improvement. Also, consider to perform some extra feature engineering, words by themselves do not provide much information. Why not trying adding n-grams & dependencies extraction, entity recognition, etc. as preprocessing steps?

[1] http://nlp.stanford.edu/projects/glove/

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.