4
$\begingroup$

I'm not a NLP guy and I have this question.

I have a text dataset containing terms which go like, "big data" and "bigdata".

For my purpose both of them are the same.

How can I detect them in NLTK (Python)?

Or any other NLP module in Python?

$\endgroup$
8
$\begingroup$

There is a nice implementation of this in gensim: http://radimrehurek.com/gensim/models/phrases.html

Basically, it uses a data-driven approach to detect phrases, ie. common collocations. So if you feed the Phrase class a bunch of sentences, and the phrase "big data" comes up a lot, then the class will learn to combine "big data" into a single token "big_data". There is a more complete tutorial-style blog post about it here: http://www.markhneedham.com/blog/2015/02/12/pythongensim-creating-bigrams-over-how-i-met-your-mother-transcripts/

$\endgroup$
  • 1
    $\begingroup$ +1 Will! I'll look at it deeper but a quick look told me it probably doesnt solve my problem. I mean what if "big data" does not happen too often? But thanks after all. Probably a good starting point! $\endgroup$ – Kasra Manshaei Jun 10 '15 at 16:50
  • 1
    $\begingroup$ It might be worth trying. It should detect at least the most common examples, and the others may not make a big difference. Also, are all of the examples you are looking for misspellings, like "bigdata"? In that case, you could compare your documents to a list of known words, and then compare them to the list of pairs of adjacent words in the text. $\endgroup$ – Will Stanton Jun 11 '15 at 1:44
  • 1
    $\begingroup$ I actually have a huge list of human generated tags and I want to make a ground-truth for them. Some people enter "big data" some enter "bigdata" and currently in my ground-truth they are two different tags but they should be identical. $\endgroup$ – Kasra Manshaei Jun 11 '15 at 15:03
  • 1
    $\begingroup$ So these are not actual sentences, just phrases without context? Another option you could try would be to use an edit distance/Levenshtein distance to group similar tags: en.wikipedia.org/wiki/Levenshtein_distance $\endgroup$ – Will Stanton Jun 11 '15 at 17:01
  • 1
    $\begingroup$ I was thinking about Levenshtein distance at the beginning but I thought maybe someone here has a better solution. Thanks for answers anyway and I'll keep this question open for a few more days to see if someone has a better idea :) $\endgroup$ – Kasra Manshaei Jun 11 '15 at 17:44
2
$\begingroup$

If you have a premade dictionary of terms, like NLTK's words.words() you can simply iterate through the string adding a space at each point and checking if they are both words. A couple possible issues come from this: 1) Compound words may be split unnecessarily and 2) tags with terms attached to compounded words would yield multiple possible results. This is where something like document distance or term frequency would come in. A simple example using a set of words called WORDS would look like:

def check_spacing(term):
    possibles = []
    for i in range(1, len(term) - 1):
        l, r = term[:i], term[i:]

        if l in WORDS and r in WORDS:
            posibbles.append((l, r))

    # probs_check is a theoretical function that returns a numeric value
    # which determines how likely each pair of words is to be what you want
    possibles = sorted(possibles, key=probs_check, reversed=True)

    return ' '.join(possibles[0])
$\endgroup$
1
$\begingroup$

Your question seems vague without the sample data you are using. How does your data-set look like? If there are delimiters within your data, you could get rid of only spaces between all words and then 'big data' & 'bigdata' would be the same, if that is what you want to do.

$\endgroup$
  • 1
    $\begingroup$ No. the point is that there are terms like "str1 str2" and "str1str2". When I want to draw histogram of course I need to count the occurrences of each term. So I need to filter the data so that all ambiguous versions of a specific term become the same. Then I start counting. $\endgroup$ – Kasra Manshaei Jun 9 '15 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.