I think Valentin's answer is great. I just wanted to +1 his remark that you really seem interested in how to find important words rather than filtering uncommon ones (bodybuilding might actually turn out to be not very common, but I understand even in that case it would still be irrelevant for your task).
If this assumption is correct, I think what you need is actually a domain classifier: you're essentially facing a classification problem where you want to classify some terms as spam or not spam. You can do that with any discriminative model if you already have a dataset or can easily create one, but even if you don't, you can do it by simply comparing two probabilities for every term in your phrases: 1) its probability over a large text file vs. 2) its probability over a smaller, domain-specific text file that you collected with a query.
Consider the data in the table below, where
- column
Term
is self-explanatory;
- column
p(term|Computer)
is the probability of term
over texts containing 3 mentions of the word computer
;
- column
p(term)
is the probability of term over all texts; (note all texts come from the same source, a large raw text dataset);
- column
domain association multiple
is the result of dividing p(term|Computer) / p(term)
;
- column
is_relevant
is boolean; if the value in domain association multiple
is larger than 10, then the word is relevant (=not common);
- (note a few words did not appear on the dataset and more data would be needed).
|Term | p(term|Computer) | p(term) | domain association multiple | is_relevant|
|---|---|---|---|---|
|server | 0.0121 | 0.0008 | 15 | 1 |
|ocr | 0 | 0 | 0 | 0 |
|mac | 0.0044 | 0.0002 | 21.4 | 1 |
|best | 0.022 | 0.0124 | 1.8 | 0 |
|data | 0.1436 | 0.023 | 6.3 | 0 |
|analysis | 0.0362 | 0.0064 | 5.6 | 0 |
|opml | 0 | 0 | 0 | 0|
|sql | 0.0027 | 0.0001 | 23.9 | 1 |
|export | 0.0016 | 0.0004 | 4.5 | 0 |
|bodybuilding | 0 | 0 | 0 | 0|
This is telling you that words server, mac
and sql
are relevant for the domain Computer
, whereas the rest are not (at the 10x threshold; a lower threshold might let analysis
and export
in, and still exclude best
).