# Text classification with thousands of output classes in Keras

I have a dataset with job titles and descriptions. The task is to predict tags for job by job title and description.

There are several tags for each job posting. Therefore, the number of labels for the model will be measured in tens of thousands.

Number of job postings = 78042

Number of unique classes (tags) = 1369

Questions:

Could you advise working types of neural networks (desirable in Keras)?

Or maybe you know how to solve this problem with the help of classical machine learning algorithms?

Still I will be very grateful for links to articles where similar problems are solved.

• Please add more data-points about the task. How many labelled examples/inputs do you have? When you say thousand output classes, is it lower end (1000 classes) or 99999 classes? Aug 21 '18 at 11:54
• @hssay, updated Aug 21 '18 at 12:33
• Acording to your histogram you have a dataset that is already labeled with tags. So this is a supervised learning problem? In other words, you want to assign tags to new untagged job postings on the basis of the existing dataset? How many tags per posting? One? 3 or 4 (the median)? Up to 15 if necessary? Up to 43?
– knb
Aug 22 '18 at 7:11
• @knb, Yes, it is a supervised learning. Median value for tags number per posting is 4. Aug 22 '18 at 7:25

The ratio of number of examples to number of classes is not large. There are few classes which have high number of occurrences (from the second graph) and the distribution seems to be following power law.

In such cases, I'll advise the following strategy,

1. Sort your tags by number of occurrences and discard tags which occur very few number of times. This will make the problem more tractable.
2. You can benchmark the accuracy that you get from classical machine learning techniques. Many classical methods support multilabel output, you can check the documentation of support in scikit-learn library here.
3. You can do a mix of unsupervised learning and nearest neighbours approach. For example, learn doc2vec embeddings on your data including tags and suggest tags from nearest matching document for a new input. The number of documents is small by doc2vec standards and you will need careful tuning of doc2vec parameters.
4. With neural networks, you can use more suitable loss functions like multilabel soft margin loss.

You could follow along how instructor Rick Scavetta processes the "Reuters newswire dataset" of short news-items into 46 news-categories (e.g. Sports-baseball). Politics/USA). I think this is very similar to your problem.

R Markdown File

• The video tutorials are currently in the making. I like the style.
• Using deep learning and the R interface to Keras, he assigns only 1 class per newsitem, not 4 or more
• Test Accuracy is 77% (but maybe you can do a better with a better lexikon)
• To my knowledge, you need a lexicon with a ranking of words (how important they are. stopwords such as "the" and super-rare words need to be ranked low)
• Does anyone know if a similar example in Python? This is exactly what I'm looking for, but I don't know R. Sep 2 '19 at 10:18

Actually, you can solve these types of problems easily with deep learning. For a moment think of a chatbot which can generate answers given questions. If we think as you mentioned each time final softmax layer should predict a probability distribution which is similar to vocabulary size. But this isn't the case. We use a loss function called Noise Contrastive Estimatimation (NCE_LOSS). Here we sample most likely words and use them to compute the softmax layer. Here I will mention Tensorflow like to understand this scenario.

• could pls share a link with the practical implementation of this algorithm (maybe, github or ipython notebook)? I just do not quite understand how to implement it. Aug 21 '18 at 12:36
• There are many. Here is one - github.com/google/seq2seq Aug 21 '18 at 16:57