1
$\begingroup$

I need to tokenize a corpus of abstracts from an international conference. The abstracts are usually American English but sometimes British English.

Consequently, I get 2 tokens for “organization” and “organisation” or “color” and “colour”. Examples : https://en.oxforddictionaries.com/spelling/british-and-spelling

Do you know a (python) library converting “British English” to “American English” (or vis versa) ?

I would be happy to that ... (but I am french and my english is not soo good)

Thanks.

$\endgroup$
1
$\begingroup$

Grouping related tokens is called text normalization.

There is not an established Python package that does this. You could create a custom dictionary or write a function to rewrite the tokens.

$\endgroup$
1
  • $\begingroup$ Thanks. I know I can make a function and what text normalization is. Just wanted to know if there is a established solution since the issue seems quite common. This dictionnary is a good start. Thanks. $\endgroup$ – user3259111 Sep 22 '17 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.