Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I already have a list of "1-gram" words, which include closed compound words like "skyscraper" or "weatherman."
However, I'm also interested in compiling a list of "2-gram" words that take on a meaning completely different than their two halves, such as "rubbber band."
Using any modules/resources/methods, how can I accomplish this?
(I tried downloading Google's Ngram files, but lots of the "2-gram" were nonsensical, like "and I" or "what was.")
A common way to frame this type of task is as collocation extraction, i.e., extracting bigrams or trigrams and then filtering them by the mutual information between their constituent tokens (this filtering solves the issue of noise you described when talking about the Google n-gram dataset). You can find more information on collocation extraction using nltk here.
Note, though, that choosing the right mutual information cut-off threshold to separate collocations from non-collocations (=n-grams) is not trivial, will require tweaking, and you will probably never be fully happy with the results :D
Generally speaking, you want it to be as high as possible, as long as that doesn't cause too many candidates to be discarded. You will probably only manage to keep about 5%-20%, which may or may not be enough for your particular needs. Note, though, that some equally-highly-collocated non-compound-noise will persist, e.g. n-grams containing verbs, phrasal verbs, some frequent adverbial phrases, and so on.
So, the options below are probably better, either on their own or combined with the information-theoretical approach (they are not mutually exclusive, for the most part):
Part of speech tagging: Run a part-of-speech tagger on your data and retain only candidate n-grams whose tokens are tagged as sequences of nouns or adjectives (NN(S) and JJ tags), or where the n-gram, at the very least, starts and ends with one such part of speech (if you're really interested only in compounds, then I think that you'll be fine with sequences like {("NN", "NN"), ("NN", "NNS"), ...} only). With this additional condition, you could lower the mutual information threshold substantially and still get informative terms.
Stopword removal: t's always easier to compile a list of all stopwords (or use/extend nltk's) than a list of all compounds, so you could also try generating all n-grams, then filtering those that i) only contain stopwords, ii) start with one, iii) or end with one. You would still end up with collocations containing verbs, so this option is not as good as #1, but it might be a working compromise.
Using gensim's phrases phrase extractor. The documentation is great so I won't expand on it here.
"Good old scraping": download a knowledge base, e.g. a dump of Wiktionary (https://en.wiktionary.org/wiki/peanut_butter) and parse it to get all dictionary lemmas a) that contain an underscore (=blank) and b) at least one dictionary entry listed as a Noun. In this way, you can filter out verbs, while at the same time validating any n-gram hypothesis as a noun against the external ground truth. Of course, this approach is limited to the coverage of the ground truth dataset (at least initially), but this may already be enough for your purposes.
For a comprehensive overview, motivation and how-to of methods #1 and #3, you can check this excellent post: https://towardsdatascience.com/word2vec-for-phrases-learning-embeddings-for-more-than-one-word-727b6cf723cf
Hope any of this helps!
