What I tried:
# -*- coding: utf-8 -*-
from nltk.stem.snowball import GermanStemmer
st = GermanStemmer()
token_groups = [(["experte", "Experte", "Experten", "Expertin", "Expertinnen"], []),
(["geh", "gehe", "gehst", "geht", "gehen", "gehend"], []),
(["gebäude", "Gebäude", "Gebäudes"], []),
(["schön", "schöner", "schönsten"], ["schon"])]
header = "{:<15} [best expected: n/n| best variants: 1/n | overlap: m]: ...".format("name")
print(header)
print('-' * len(header))
for token_group, different_tokens in token_groups:
stemmed_tokens = [st.stem(token) for token in token_group]
different_tokens = [st.stem(token) for token in different_tokens]
nb_expected = sum(1 for token in stemmed_tokens if token == token_group[0])
nb_variants = len(set(stemmed_tokens))
overlap = set(stemmed_tokens).intersection(set(different_tokens))
print("{:<15} [as expected: {}/{}| variants: {}/{} | overlap: {}]: {}".format(token_group[0], nb_expected, len(token_group), nb_variants, len(token_group), len(overlap), stemmed_tokens))
what I get:
experte [as expected: 0/5| variants: 3/5 | overlap: 0]: ['expert', 'expert', 'expert', 'expertin', 'expertinn']
geh [as expected: 3/6| variants: 4/6 | overlap: 0]: ['geh', 'geh', 'gehst', 'geht', 'geh', 'gehend']
gebäude [as expected: 0/3| variants: 1/3 | overlap: 0]: ['gebaud', 'gebaud', 'gebaud']
schön [as expected: 0/3| variants: 1/3 | overlap: 1]: ['schon', 'schon', 'schon']
The two main problems are:
- Overlaps: schön != schon
- Non-working stemming, e.g. [experte, expertin, expertinnen], [ich gehe, du gehst, er geht]
A not so serious side problem is matching my expectations. So if the stemmer could actually bring the word in a basic form (not simply the stem), then it would be easier to analyze.
More Examples
Clashes
- Input -> Output != clash
- mittels -> mittel != "Das Mittel"
Unmatched expectations
Input -> Output / expected
Mädchen -> madch / Mädchen
- Behaarung -> behaar / Behaarung