I'm not a NLP guy and I have this question.
I have a text dataset containing terms which go like, "big data" and "bigdata".
For my purpose both of them are the same.
How can I detect them in NLTK (Python)?
Or any other NLP module in Python?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
There is a nice implementation of this in gensim: http://radimrehurek.com/gensim/models/phrases.html
Basically, it uses a data-driven approach to detect phrases, ie. common collocations. So if you feed the Phrase class a bunch of sentences, and the phrase "big data" comes up a lot, then the class will learn to combine "big data" into a single token "big_data". There is a more complete tutorial-style blog post about it here: http://www.markhneedham.com/blog/2015/02/12/pythongensim-creating-bigrams-over-how-i-met-your-mother-transcripts/
If you have a premade dictionary of terms, like NLTK's
words.words() you can simply iterate through the string adding a space at each point and checking if they are both words. A couple possible issues come from this: 1) Compound words may be split unnecessarily and 2) tags with terms attached to compounded words would yield multiple possible results. This is where something like document distance or term frequency would come in. A simple example using a set of words called
WORDS would look like:
def check_spacing(term): possibles =  for i in range(1, len(term) - 1): l, r = term[:i], term[i:] if l in WORDS and r in WORDS: posibbles.append((l, r)) # probs_check is a theoretical function that returns a numeric value # which determines how likely each pair of words is to be what you want possibles = sorted(possibles, key=probs_check, reversed=True) return ' '.join(possibles)
Your question seems vague without the sample data you are using. How does your data-set look like? If there are delimiters within your data, you could get rid of only spaces between all words and then 'big data' & 'bigdata' would be the same, if that is what you want to do.