Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have a huge list of short phrases, for example:
sql server data analysis # SQL is not a common word
bodybuilding # common word
export opml # opml is not a common word
best ocr mac # ocr and mac are not common words
I want to detect if word is not a common word and should not be processes further.
I've tried to do this with NLTK, but it gives strange results:
result = word in nltk.corpus.words.words()
sql = false
iso = true
mac = true
Is there a better way to do this?
It all depends on your definition of what a common word is in your domain. You are using an NLTK corpus which likely doesn't fit your domain very well. Either you have a corpus containing the domain you want and you do a simple lookup. Or you don't know in advance and you need to compute these common words from your documents (your short phrases). In that case, the more sentences you have the better.
An easy way to do this is using pure Python is to use a counter
from collections import Counter
documents = [] # here add your list of documents/phrases
counter = Counter()
for doc in documents:
words = doc.split() # assuming that words can be split on whitespaces
counter.update(words)
counter.most_common() # this will return words ranked by their frequency
Then it's up to you to apply a threshold to define what words are common and which aren't. A more advanced approach could be using their TFIDF weights but in that case, you are not necessarily keeping common words, you are keeping "important" ones.
Some common approaches to this problem are:
I think Valentin's answer is great. I just wanted to +1 his remark that you really seem interested in how to find important words rather than filtering uncommon ones (bodybuilding might actually turn out to be not very common, but I understand even in that case it would still be irrelevant for your task).
If this assumption is correct, I think what you need is actually a domain classifier: you're essentially facing a classification problem where you want to classify some terms as spam or not spam. You can do that with any discriminative model if you already have a dataset or can easily create one, but even if you don't, you can do it by simply comparing two probabilities for every term in your phrases: 1) its probability over a large text file vs. 2) its probability over a smaller, domain-specific text file that you collected with a query.
Consider the data in the table below, where
Term is self-explanatory;p(term|Computer) is the probability of term over texts containing 3 mentions of the word computer;p(term) is the probability of term over all texts; (note all texts come from the same source, a large raw text dataset);domain association multiple is the result of dividing p(term|Computer) / p(term);is_relevant is boolean; if the value in domain association multiple is larger than 10, then the word is relevant (=not common); |Term | p(term|Computer) | p(term) | domain association multiple | is_relevant|
|---|---|---|---|---|
|server | 0.0121 | 0.0008 | 15 | 1 |
|ocr | 0 | 0 | 0 | 0 |
|mac | 0.0044 | 0.0002 | 21.4 | 1 |
|best | 0.022 | 0.0124 | 1.8 | 0 |
|data | 0.1436 | 0.023 | 6.3 | 0 |
|analysis | 0.0362 | 0.0064 | 5.6 | 0 |
|opml | 0 | 0 | 0 | 0|
|sql | 0.0027 | 0.0001 | 23.9 | 1 |
|export | 0.0016 | 0.0004 | 4.5 | 0 |
|bodybuilding | 0 | 0 | 0 | 0|
This is telling you that words server, mac and sql are relevant for the domain Computer, whereas the rest are not (at the 10x threshold; a lower threshold might let analysis and export in, and still exclude best).
