So I know there are many methods to classify sentences into types. Like in sentiment analysis (positive, negative, neutral), spam emails (spam, not spam), etc. The thing I want to ask is how would I find the words most responsible for the categorisation. For example: sad-negative, happy-positive, the-no information, are-no information.

So how do I find the words which impact the classification?

  • $\begingroup$ Do you want to show the impact of a word in general? Or do you want to see the impact of a specific word on a specific sentence? $\endgroup$ Jul 5, 2018 at 11:50

1 Answer 1


First of all, let's just clarify that proving causation is quite difficult. And therefore, you will mostly need to show correlation. Also, for text classification, the impact of each word is not linear. Each word impacts the meaning of the sentence relative to the other words and vice versa, therefore, you need to keep that in mind.

With that said, there are a few ways you can look at the impact of words on a classification:


If you have multiple classes, you could look at how frequently certain words appear in each class. If the word "good" only appears in positive sentences, it is safe to assume that it has a high impact.

This method has the benefit of being model-independent.

Feature importance

If you are using a bag-of-word representation as your input, you could look at how much the presence/absence of each feature affects the results. For instance, let's say that you can classify positive sentences with 80% accuracy with all words as features. Now, try to classify the sentences again by removing certain words. If you remove the word "good" from sentences, you can look at how the performance varies.

Or, you could do the opposite and try to simply classify each word. If you classify the word "good" as positive, it means that it is probably a word that impacts sentences into being positive. If your model provides confidence, you could use that to know how much information the word brings to the sentence.

Attention mechanism

The attention mechanism will be able to tell you which part of the input influences the results the most. This works better if you use a sequential model

  • $\begingroup$ Fair enough I don't know about attention mechanism but I'll learn about it...The rest are infeasible in my problem, because I am doing genetic analysis where words appear many times, multiple times, in short can't be dome $\endgroup$
    – DuttaA
    Jul 5, 2018 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.