Algorithms and tools for ranking text as a job description

Question

We're working on a ranking texts by degree of the similarity to vacancies. We have 4-year data set (≈1M texts) of custom search feeds from social network. We also have vacancies (≈30K) manually selected from this set.

Now the vacancies are selected in 2 stages:

1. We make several specific search queries (i.e. "need translator") on the social network by API, receiving and combining the search feeds into one.

2. Then we look through each post in it for real vacancies. Generally, it should contain a request for interpreter services. Often it contains some job details (such as a place, salary, subject, deadlines, languages, contacts) and stylistic features of speech (i.e. greetings, exclamations, request to write in private, ask for recommendations, etc.).

Before the point 2, we would like to have first in our final moderation feed the texts, that are most likely to be vacancies. We're going to try ranking by using the Naive Bayes classifier or the Natural Language Processing, but it's not as simple as it seems: there are a lot of various tools, schemes, models.

The question is, what algorithms and tools could help in our case? What we should pay attention to?

Answer 1

If I understand the problem correctly, there are several approaches that can be taken.

We'll begin with the least deep learning oriented solution first and slowly move into the spectrum of Deep Learning. This choice is primarily for cost and difficulty reasons. The amount of bugs that you can find with a deep learning solution can be very difficult to understand in certain situations and that results in slower prototyping and ultimately slower time to production.

At the very base, your problem deals with ranking the jobs with respect to how relevant they are to a particular job. The literature for ranking problems is vast and I will suffix my answer with links to these papers or studies. I will also say that instead of dealing with this as a ranking problem there may be another way of going about it. However, this is hinged on the type of data you possess. If you know that each query is certainly looking for a particular type of job. Then, you can break this down into a multi-task problem, where one of the tasks is figuring out whether there is a vacancy or not and the second is figuring out if the vacancy relates in any way to the search query.

Solution 1: Take the dataset and handcraft a set of features that you think are relevant to the process of classifying the output. To do this effectively, you should know certain core features that you think relate the input features to the final label (here I use input features to represent the features that you think represent your features well. For example, some features that may be relevant is information about the person or entity posting the title (in stage 2), if they are a well known company in the domain (this can be accesses through some intelligent web scraping and text retrieval methods). Then, it is possibly more likely for it to be a posting, right? Let us assume now that you have constructed this set of features. Now, for each of the set of features, you have a corresponding label to whether there is a vacancy or not. To turn this approach into a ranking problem is very simple. All you have to do is observe the confidence (the softmax probability is good enough) and the one which has a higher probability is ranked higher). This solution is an easy one to accomplish using even simple feed forward neural networks. You would also benefit from testing out simple classifiers like SVMs or Random Forests. It must also be remembered that the actual sentence must be embedded into some numeric space, like a sequence of vectors in $R^n$. This solution does not detail all of the explicit steps but provides a general framework with which you should be able to attack the problem.

Drawbacks The above method requires a lot of time spent in handcrafting the feature set and there is going to be a lot of iteration because the feature set is never going to be perfect. There is always going to be something that you possibly missed out. Secondly training a good word embedding is not a very easy task. You may have to spend a lot of time getting that running, especially if you want the embedding to be somehow related to your words.

Solution 2 This solution begins to touch the fence of Deep Learning. This solution is a comparison of the attention encodings within the source utterance and the target utterance. The source utterance is, a phrase like "need janitor". Now you should train an attention model over the sequence of words and observe what is the word(s) being given the maximum attention to. Once you know this, the problem becomes a question of consolidating the attentions in some meaningful manner and comparing the distance between the source and target through some metric space. So you should certainly pay attention to attention based models (haha!). Again, there are going to be hiccups in the process of figuring out how much attention actually affects the output and whether it is a good model for your data. Distances between attention will return a vector and you can consider the $\mathcal{L}_2$ norm or any other map from $\mathbb{R^n} \rightarrow \mathbb{R}$.

Drawbacks Attention distances are a good metric, however, they are extremely data centric. For example if the ads are very long, it may be true that the model does not pick up the appropriate words to attend to.

Now we get to some resources that may be useful to go through in order to understand exactly how the different components of the answer work.

1. https://en.wikipedia.org/wiki/Learning_to_rank
2. http://www.shivani-agarwal.net/Events/SDM-10-Tutorial/sdm10-tutorial.pdf
3. https://towardsdatascience.com/learning-to-rank-with-python-scikit-learn-327a5cfd81f
4. http://times.cs.uiuc.edu/course/598f16/l2r.pdf
5. https://www.stat.uchicago.edu/~lekheng/meetings/mathofranking/slides/agarwal.pdf

Now for the posts about attention:

1. http://www.wildml.com/2016/01/attention-and-memory-in-deep-learning-and-nlp/

2. http://akosiorek.github.io/ml/2017/10/14/visual-attention.html

3. https://medium.com/syncedreview/a-brief-overview-of-attention-mechanism-13c578ba9129

Finally, my last component of the answer will deal with evaluating exactly how well you have done using metrics. I have added some metrics above in my Answer, however there is another metric which in particular represents the similarity, syntactically between the input query and the output query.

1. https://en.wikipedia.org/wiki/BLEU

Answer 2

My suggestion would be a sequence classification approach using Deep Learning, in particular Recurrent Neural Networks, which are proven to be powerful in text processing. For fast prototyping, you should use the Keras API (with Tensorflow backend).

In short:

1. Label your dataset (2 classes: vacancies or not-vacancies)
2. Embed every word in your dataset into a vector of integers, using the Embedding layer of Keras
3. Find the "X" most common words in your dataset, where "X" depends on the size of your dataset
4. Constrain your dataset to include only those words and remove the rest
5. Pad or truncate the sequences to have the same length using keras.preprocessing.sequence.pad_sequences()
6. Train an LSTM-RNN on your labeled dataset (simply by inserting a LSTM layer in Keras) and then test it

Extra tips for improvement: use a Dropout layer to decrease the variance of your model and also try improving it by combining it with a Convolutional Neural Network (CNN) to exploit spatial relations in the data. This can be easily implemented if you insert a Conv1D layer before the LSTM layer.

An excellent reference that I have found online is this.