# Text post-processing

I have set of newspaper articles and I use TextRank algorithms to identify their keywords to perform a classification.

Apart from the important informative keywords, I am also receiving garbage keywords as follows.

"viewed", "flutter", "function", "k", "neighbour"


Is there a way to post process and remove these keywords?

Yes, you can do that by add them to the existing NLTK Stop-word dictionary for all such words/Creating a Custom Stop-word dictionary.

for Custom Stop-word dictionary you need to include all the key words in the dictionary before processing, make sure to check if the words are present in the stop-word dictionary. If yes, remove them from the text else nothing needs to be done.

1. set of code is for updating new words to the NLTK Stop-words dictionary:

#Custom function to remove the stop-words def removeStopWords(str): #select english stopwords from NLTK cachedStopWords = set(stopwords.words("english")) #add custom words, mentioned above cachedStopWords.update(("viewed", "flutter", "function", "k", "neighbour")) #remove stop words new_str = ' '.join([word for word in str.split() if word not in cachedStopWords]) return new_str

2. set of code is for making a Custom stop-word dictionary:

stopwordList = ["viewed", "flutter", "function", "k", "neighbour"] StopWordsRemover(inputCol="words", outputCol="filtered" ,stopWords=stopwordList)

• Thank you for your answer :) The problem I have is I don't know these garbage words before hand. Is there a way to detect these meaningless words? – Volka Nov 29 '17 at 5:27
• As far as I know, you cannot. To overcome such issue I would do couple of test iterations and extract the words which ever are unnecessary. Add them to my stop word list and when I do run on actual data I would remove all the stop words. The reason why i think it is not possible is every word in different domains mean different, generalizing them is not going to be an easy task. – Toros91 Nov 29 '17 at 5:33
• I have been working on something similar and have realized that a little bit, if not more, of manual cleaning would be required. – trollster Nov 29 '17 at 15:12
• @trollster yeah it would take couple of iterations to automate as you said it would take time and patience to figure out. – Toros91 Nov 29 '17 at 15:41

Identifying uninformative words is not an easy task and is domain-dependent. For example, stop words or punctuation often are discriminative a lot for sentiment analysis.

If you want to test the keyness of a words for classification, you may use selection scores as chi2, information gain or mutual information. chi2 score basically assess how good is a score at discriminating two classes based on occurences count.

Thus you could remove the lowest scored words with this technique. Post-processing can be done as explained by @Toros91, adding that if for instance you consider characters to be garbage you can easily prior remove them based on string length.

Finding the "garbageness" of words is I think not an easy problem and is often over-simplified in text classification.

Don't hesitate if you have further questioning.

Using scikit-learn :

from sklearn.datasets import load_iris
from sklearn.feature_selection import SelectPercentile
from sklearn.feature_selection import chi2
import numpy

X, y = iris.data, iris.target
selector = SelectPercentile(score_func=chi2, percentile=100)
X_reduced = selector.fit_transform(X, y)
indices = numpy.argsort(selector.scores_)
# array containing the indices of features according to ascendant chi2 score


You can then use the indices to remove k worst feature according to chi2 square in your data.

• Thanks a lot for your answer :) Can you please provide me a simple example of chi2 (in python)? – Volka Nov 29 '17 at 8:27
• I added it in the answer :) – Elliot Nov 29 '17 at 11:25
• As fair I as know, the three mesures can approximately be derived from each other and thus should produce quite the same results. Though, it depends again on the data you have. In my experience, chi2 performs faster than mutual information. Moreover, I haven't heard about information gain implemented in feature selection of scikit. – Elliot Nov 29 '17 at 12:39
• You're welcome. The answer is sadly no, what values would you have for the words ? – Elliot Nov 30 '17 at 3:45
• Okay so it's good for selection methods then. If you want to filter on frequency count, you can use the min_df argument of vectorizers in scikit learn. – Elliot Nov 30 '17 at 5:06