2
$\begingroup$

I am classifying text using fastText which is a word2vec library that can also create vectors for character level n-grams and I have successfully trained a binary classifier.

Now I’d like to see what words or subword n-grams are the most predictive of a class for the two classes (e.g. if classifier sees a word forest or a subword res then that might be a strong indication that the document has label Nature, but if it sees word “and” then that is probably not very informative for this classification task).

Therefore, I guess the question could be phrased as:

Given vectors representing words and subwords and a trained fastText classifier, what would be the best way to get a list of e.g. top 10 most informative words and subwords for deciding which class a sample belongs to?

Even though I’d be glad if you could make specific suggestions that consider my current setup with fastText, I’m also open to the more general solution suggestions.

Thanks

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

Note 1: I believe the technical term for what you are looking for is '(sub)word salience'.

First thought (which is not the best method, but could be worth a quick try if your examples are smallish) - Run the text with all words through fastText, and get the baseline most likely labels with probabilities (predict-prob function). Then remove each word that you are interested in testing, (maybe removing common words like 'a' 'the' etc), and compare the predicted probabilities of each obtained set of predicted_probs. The ones that give the greatest difference between baseline and missing wordX can be interpreted as the most informative.

A permutation of this idea is to get the embedding for the interesting words, and then get a distance measure (I assume cosine distance) to the labels you have. Then you have a problem, as they may be distributed evenly across labels in terms of distance, but if there is a group of words that are close to a particular label, that can be interpreted as informative.

I am sure there is a better technique to do this, so I am curious as well :). You may want to check these out (but they use additional modeling to get salience)

Learning Neural Word Salience Scores.

A NEW METHOD OF REGION EMBEDDING FOR TEXT CLASSIFICATION - this deals with the exact problems of how to identify the useful bits of an input text

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the ideas and the links. In the meantime I also thought of calculating TF-IDF for each class and then comparing the scores between the same words from the two classes. Score deviating from zero implies that the word is specific for given class while scores close to zero probably indicate that these words are not very helpful with classification. I'm not sure if I explained it well, but what do you think about this idea? $\endgroup$
    – leopik
    Apr 28, 2018 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.