Is there a good German Stemmer?

Question

What I tried:

# -*- coding: utf-8 -*-

from nltk.stem.snowball import GermanStemmer
st = GermanStemmer()

token_groups = [(["experte", "Experte", "Experten", "Expertin", "Expertinnen"], []),
                (["geh", "gehe", "gehst", "geht", "gehen", "gehend"], []),
                (["gebäude", "Gebäude", "Gebäudes"], []),
                (["schön", "schöner", "schönsten"], ["schon"])]
header = "{:<15} [best expected: n/n| best variants: 1/n | overlap: m]: ...".format("name")
print(header)
print('-' * len(header))
for token_group, different_tokens in token_groups:
    stemmed_tokens = [st.stem(token) for token in token_group]
    different_tokens = [st.stem(token) for token in different_tokens]
    nb_expected = sum(1 for token in stemmed_tokens if token == token_group[0])
    nb_variants = len(set(stemmed_tokens))
    overlap = set(stemmed_tokens).intersection(set(different_tokens))
    print("{:<15} [as expected: {}/{}| variants: {}/{} | overlap: {}]: {}".format(token_group[0], nb_expected, len(token_group), nb_variants, len(token_group), len(overlap), stemmed_tokens))

what I get:

experte  [as expected: 0/5| variants: 3/5 | overlap: 0]: ['expert', 'expert', 'expert', 'expertin', 'expertinn']
geh      [as expected: 3/6| variants: 4/6 | overlap: 0]: ['geh', 'geh', 'gehst', 'geht', 'geh', 'gehend']
gebäude  [as expected: 0/3| variants: 1/3 | overlap: 0]: ['gebaud', 'gebaud', 'gebaud']
schön    [as expected: 0/3| variants: 1/3 | overlap: 1]: ['schon', 'schon', 'schon']

The two main problems are:

• Overlaps: schön != schon
• Non-working stemming, e.g. [experte, expertin, expertinnen], [ich gehe, du gehst, er geht]

A not so serious side problem is matching my expectations. So if the stemmer could actually bring the word in a basic form (not simply the stem), then it would be easier to analyze.

More Examples

Clashes

• Input -> Output != clash
• mittels -> mittel != "Das Mittel"

Unmatched expectations

• Input -> Output / expected

• Mädchen -> madch / Mädchen

• Behaarung -> behaar / Behaarung

Answer 1

Big problem and very good question!

I used spacy in the past, which has a German module. I guess stemming is not supported, but lemmatization.

Looking at the output below, I don't think that spacy will solve your problem to be honest. However, I just wanted to let you know about this option.

Spacy Lemmatization:

#pip install spacy
#python -m spacy download de

import spacy
nlp = spacy.load('de_core_news_sm')

mywords = "Das ist schon sehr schön mit den Expertinnen und Experten"

for t in nlp.tokenizer(mywords):
    print("Tokenized: %s | Lemma: %s" %(t, t.lemma_))

Result:

Tokenized: Das | Lemma: der
Tokenized: ist | Lemma: sein
Tokenized: schon | Lemma: schon
Tokenized: sehr | Lemma: sehr
Tokenized: schön | Lemma: schön
Tokenized: mit | Lemma: mit
Tokenized: den | Lemma: der
Tokenized: Expertinnen | Lemma: Expertinnen
Tokenized: und | Lemma: und
Tokenized: Experten | Lemma: Experte

Answer 2

The question is already almost 2 years old, but I guess many people struggle with the same question.

Many people use the TreeTagger for this. The TreeTagger does POS-Tagging and limmatization, but you need to install the TreeTagger by hand (but that is very easy to do) and than install a Python-wrapper if you use Python:

import treetaggerwrapper
import nltk
from pprint import pprint

tree_tagger = treetaggerwrapper.TreeTagger(TAGLANG='de')

sent = "Das ist schon sehr schön mit den Expertinnen und Experten."

words = nltk.word_tokenize(sent)
tags = tree_tagger.tag_text(words,tagonly=True) #don't use the TreeTagger's tokenization!
nice_tags = treetaggerwrapper.make_tags(tags)
pprint(nice_tags)

This gives the following result:

[Tag(word='Das', pos='PDS', lemma='die'),
 Tag(word='ist', pos='VAFIN', lemma='sein'),
 Tag(word='schon', pos='ADV', lemma='schon'),
 Tag(word='sehr', pos='ADV', lemma='sehr'),
 Tag(word='schön', pos='ADJD', lemma='schön'),
 Tag(word='mit', pos='APPR', lemma='mit'),
 Tag(word='den', pos='ART', lemma='die'),
 Tag(word='Expertinnen', pos='NN', lemma='Expertin'),
 Tag(word='und', pos='KON', lemma='und'),
 Tag(word='Experten', pos='NN', lemma='Experte'),
 Tag(word='.', pos='$.', lemma='.')]

As an alternative you could use HanTa (https://github.com/wartaal/HanTa):

!pip install HanTa
from HanTa import HanoverTagger as ht
import nltk
from pprint import pprint

tagger = ht.HanoverTagger('morphmodel_ger.pgz')

sent = "Das ist schon sehr schön mit den Expertinnen und Experten."

words = nltk.word_tokenize(sent)
lemmata = tagger.tag_sent(words,taglevel= 1)
pprint(lemmata)

For the given sentence this gives basically the same result as before:

[('Das', 'das', 'PDS'),
 ('ist', 'sein', 'VAFIN'),
 ('schon', 'schon', 'ADV'),
 ('sehr', 'sehr', 'ADV'),
 ('schön', 'schön', 'ADJD'),
 ('mit', 'mit', 'APPR'),
 ('den', 'den', 'ART'),
 ('Expertinnen', 'Expertin', 'NN'),
 ('und', 'und', 'KON'),
 ('Experten', 'Experte', 'NN'),
 ('.', '--', '$.')]

Finally, you could use a tool called GermaLemma, which, however, requires that you run another tool for POS-tagging before.

Answer 3

NLTK has an implementation of a stemmer specifically for German, called Cistem. I think it was added with NLTK version 3.4.
While the results on your examples look only marginally better, the consistency of the stemmer is at least better than the Snowball stemmer, and many of your examples are reduced to a similar stem.

There is still the removal of Umlauts, as well as a reduction of word stems (especially prefixes such as ge- are sometimes removed as well), which might not match your expectations. The results below are from a run with Cistem(case_insensitive=True).

name            [best expected: n/n| best variants: 1/n | overlap: m]: ...
--------------------------------------------------------------------------
experte         [as expected: 0/5| variants: 4/5 | overlap: 0]: ['exper', 'expert', 'expert', 'experti', 'expertinn']
geh             [as expected: 5/6| variants: 2/6 | overlap: 0]: ['geh', 'geh', 'geh', 'geh', 'geh', 'hend']
gebäude         [as expected: 0/3| variants: 1/3 | overlap: 0]: ['baud', 'baud', 'baud']
schön           [as expected: 0/3| variants: 2/3 | overlap: 1]: ['schon', 'schoner', 'schon']

EDIT: I should also add that the implementation is derived form the work by Leonie Weissweiler and Alexander Fraser, who also compile a very nice comparison of various stemmers in their paper.