4
$\begingroup$

I am new to practicing NLP and most topics related, but I want to make a program that can gather and extract data for me on its own.

To be more specific, I want to tell the program "I want more information on this topic(i.e heart attacks)", and then the program shall find, gather and extract meaningful texts on the topic from around the www.

I happen to live in Norway, which means that most interesting data will be in English, but I also want to fetch interesting data found in Norwegian.

One challenge is the differences in stop words. For instance, "are" and "and" are both stop words in English and subjects in Norwegian. Other challenges are also likely to occur.

So my question is: Would I need to create separate algorithms for every natural language to be interpreted?

$\endgroup$
5
$\begingroup$

There are a few ways to deal with this issue. Python has a package called NLTK which contains stop word lists for several languages (including English and Norwegian). You can simply use this package, it's usage is as follows:

>>> from nltk.corpus import stopwords
>>> stop = stopwords.words('english')
>>> sentence = "this is a foo bar sentence"
>>> print [i for i in sentence.split() if i not in stop]
  ['foo', 'bar', 'sentence']

Alternatively, a method for automatically suppressing stop words is called tf-idf; tf-idf is commonly used in search engines so that the most important words are promoted to the forefront. In your case, I would suspect you'd want to have IDF scores for both English and Norwegian and apply only the appropriate one on a language to language basis.

http://en.wikipedia.org/wiki/Tf%E2%80%93idf

$\endgroup$
2
$\begingroup$

So my question is: Would I need to create separate algorithms for every natural language to be interpreted?

Yes, I believe so.

But building a model for detecting the used language is not hard: usually taking n-grams (n-shingles) and then doing classification on them works quite well in practice. By the way, for the start you can use stop words to detect a language, e.g. like it's described here.

Then once the language is detected, I'd do the NLP stuff for each language separately.

$\endgroup$
1
$\begingroup$

If I understood your question correctly, you want to be able to extract keywords from texts in different languages. One thing for sure, you will need a list of stopwords for each language. As someone else mentioned, that can be also obtained with TF-IDF. Algorithm for extracting keywords that I came across and seems promising is ToPMine - http://web.engr.illinois.edu/~elkishk2/papers/ToPMine.pdf It is fairly well explained in this text.

Hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.