3
$\begingroup$

I'm quite new to the NLTK package of Python and to NLP too (I usually work in R but for NLP purposes and scraping maybe Python is more able).

I scrap articles from Hungarian newsportals and want to make a wordcloud out of it to show what are the current trending news topics. First I filter out stopwords and then stem the remaining words. (nltk has Hungarian stemmer) So I'm able to make a frequency table which can be the base of the wordcloud. My problem comes afterwards because stems are usually meaningless chunks (and not lemmas) of real words. I want to somehow complete the stem to a real word.

My first idea was to assign the most common word or the shortest one (or some combination of this 2 rules) to the stem and represent that in the wordcloud.

Is there a better solution for stem completion or should I follow a different workflow?

Improve this question
$\endgroup$
3
  • $\begingroup$ Few clarifications: Are you having problems with the stems that are generated, just asking if their is a lemmatisation module that handles Hungarian, or are you asking which algorithm you should use to classify and build a wordcloud after stem creation? Or am I completely missing your question? $\endgroup$
    – LinkBerest
    Jun 12, 2015 at 21:25
  • $\begingroup$ Stems are good to identify different forms of the same word.But stems are not actual words usually just word chunks(at least in Hungarian). I want to somehow translate stems back to real words that I can use later in my wordcloud. Lemmatisation would be nice,it would handle the problem, but I did not find a Hungarian lemmatisation module (and because of the complexity of the Hungarian language I think it is very difficult to create one). So I want a solution to replace the stems with one of the real word occurrences in the text. Or any other workflow is welcome that would solve my problem. $\endgroup$
    – Viktor
    Jun 12, 2015 at 22:56
  • $\begingroup$ Perhaps this helps github.com/oroszgy/awesome-hungarian-nlp (I guess this is has been developed in the meantime). PS: How many words forms are there roughly in your data set? I'm estimating that you could easily maintain a dict in English (German, French, Dutch, for that matter). But if there are many inflections in the language that might be difficult. $\endgroup$
    – S van Balen
    May 6, 2020 at 15:17

1 Answer 1

Reset to default
0
$\begingroup$

In order to preserve the mapping, you will have to store both the original text and the stemmed version in the frequency table. The frequency counts will be on the stemmed version. The display version will the be the set of original tokens associated with a given stem.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.