3
$\begingroup$

Sentencer from SpaCy and NLTK does not catch the fact that typical abbreviations (e.g. Mio. for Million in German) and the resulting sentence split is not correct. I understand that sentencers are supposed to be simple and quick but I am wondering if there is a better one that takes into account something more than uppercased words and punctuation? Alternatively, how to make SpaCy / NLTK / ... sentencer work for such sentences?

I am interested primarily with sentencers with Python API.

$\endgroup$
2
$\begingroup$

Neural tools trained on Universal Dependencies corpora use learned models for tokenization and sentence-spliting. Two I know of are:

  • UDPipe – developed at Charles University in Prague. Gets very good results (at least for parsing), but has a little unintuitive API.

  • Stanza – developed at Stanford University. The API is quite similar to Spacy.

However, they are quite slow compared to regex-based sentence-spliting.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the references. The speed is not an issue. $\endgroup$ – sophros Oct 13 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.