Detect if word is «common English» word or slang word

Question

I have a huge list of short phrases, for example:

sql server data analysis # SQL is not a common word
bodybuilding # common word
export opml # opml is not a common word
best ocr mac # ocr and mac are not common words

I want to detect if word is not a common word and should not be processes further.

I've tried to do this with NLTK, but it gives strange results:

result = word in nltk.corpus.words.words()

sql = false
iso = true
mac = true

Is there a better way to do this?

Answer 1

It all depends on your definition of what a common word is in your domain. You are using an NLTK corpus which likely doesn't fit your domain very well. Either you have a corpus containing the domain you want and you do a simple lookup. Or you don't know in advance and you need to compute these common words from your documents (your short phrases). In that case, the more sentences you have the better.

An easy way to do this is using pure Python is to use a counter

from collections import Counter

documents = [] # here add your list of documents/phrases

counter = Counter()

for doc in documents:

    words = doc.split() # assuming that words can be split on whitespaces

    counter.update(words)

counter.most_common() # this will return words ranked by their frequency

Then it's up to you to apply a threshold to define what words are common and which aren't. A more advanced approach could be using their TFIDF weights but in that case, you are not necessarily keeping common words, you are keeping "important" ones.

Answer 2

Some common approaches to this problem are:

• Keep only the n- most common words in a corpus (automatically done in scikit-learn: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html#sklearn.feature_extraction.text.CountVectorizer).
• Keep all words, but downweight uninformative words using a transformation called tf-idf scaling https://en.wikipedia.org/wiki/Tf%E2%80%93idf
• Manually remove a list of "stopwords". For a recently published (automatically generated!) list of english language stopwords see https://github.com/amarallab/stopwords/blob/master/data/s2019-05-29_stopword-corpora/stopword_list_en

Answer 3

I think Valentin's answer is great. I just wanted to +1 his remark that you really seem interested in how to find important words rather than filtering uncommon ones (bodybuilding might actually turn out to be not very common, but I understand even in that case it would still be irrelevant for your task).

If this assumption is correct, I think what you need is actually a domain classifier: you're essentially facing a classification problem where you want to classify some terms as spam or not spam. You can do that with any discriminative model if you already have a dataset or can easily create one, but even if you don't, you can do it by simply comparing two probabilities for every term in your phrases: 1) its probability over a large text file vs. 2) its probability over a smaller, domain-specific text file that you collected with a query.

Consider the data in the table below, where

1. column Term is self-explanatory;
2. column p(term|Computer) is the probability of term over texts containing 3 mentions of the word computer;
3. column p(term) is the probability of term over all texts; (note all texts come from the same source, a large raw text dataset);
4. column domain association multiple is the result of dividing p(term|Computer) / p(term);
5. column is_relevant is boolean; if the value in domain association multiple is larger than 10, then the word is relevant (=not common);
6. (note a few words did not appear on the dataset and more data would be needed).

    |Term | p(term|Computer) | p(term) | domain association multiple | is_relevant|
    |---|---|---|---|---|
    |server | 0.0121 | 0.0008 | 15 | 1 |
    |ocr | 0 | 0 | 0 | 0 |
    |mac | 0.0044 | 0.0002 | 21.4 | 1 |
    |best | 0.022 | 0.0124 | 1.8 | 0 |
    |data | 0.1436 | 0.023 | 6.3 | 0 |
    |analysis | 0.0362 | 0.0064 | 5.6 | 0 |
    |opml | 0 | 0 | 0 | 0|
    |sql | 0.0027 | 0.0001 | 23.9 | 1 |
    |export | 0.0016 | 0.0004 | 4.5 | 0 |
    |bodybuilding | 0 | 0 | 0 | 0|

This is telling you that words server, mac and sql are relevant for the domain Computer, whereas the rest are not (at the 10x threshold; a lower threshold might let analysis and export in, and still exclude best).