Just landed on a Entity Resolution (Matching) project. I am working in python and I was wondering if there's an off-the-shelf implementation of string harmonization functions, including:

  • removal of trailing spaces
  • removal of multiple spaces
  • removal of special characters (eg. ü)
  • ...

All this is easily done with string and Regex operations but I do not want to reinvent the wheel and tap into something already ready (and perhaps learn from that, shall I need to move on to something custom later). Are there readily available libs for that? Tried nltk, which suggests the use of regex expression, textblob, and googling around but found nothing really helpful so far - probabily because I did not get to use the right keyword.

As an example, it would be useful to have a function that replaces special characters from foreign languages with morphological equivalents - for example "ß" in German with "ss" ("straße" -> "strasse")

Edit: clarification In my question, I am not really specifying the problem at hand. This is because what I am trying to get to is really a "grammar" that would let me and the other ppl in my team define workflows of harmonization/canonicalisation operations.

I am at the moment Working on this as a chain of functions f: string -> string. Whether some steps are part of it, it will depend on the problem at hand. Workflow will be likely tested empirically (we'll pick the workflow that leads to the best performing algo)

Note: this is more frequently called canonicalisation than harmonisation. URLify and URL-safe are also terms that can help you in your search.

whitespace
Combining the built-in functions string.split + string.join is a solid way to canonicalise intermediate and trailing whitespace, including tabs, carriage returns and so on.

non-ASCII chars
ASCIIification is actually somewhat contrary to canonicalisation. That is, you should think hard about whether you really want to be converting 'München' to 'muenchen', rather than 'muenchen' to proper 'München'.

If you do want to go in that direction, you should give more details on the use case and the desired conversions. Here are some examples of possible input:

Where is Münsterstraße?
ısparta'daki istanbul'dakı ve gülenciler!
Isparta'daki İstanbul'dakı ve Gülenciler!
¿Cómo está?
Это вообще фарс...
Биће белајааа, чорба
Biće belajaaa, čorba
You can now resume your résumé.

Converting to ASCII will be lossy and there are some decisions to make, often dependent on the source and target language.

If you define the likely input, scope and desired behaviour then others can make recommendations.

  • Thanks. Well, what I am trying to get to is really a "grammar" that would let me and the other ppl in my team define workflows of canonicalisation operations. Working on this as a chain of functions f: string -> string. Whether some steps are part of it, it will depend on the problem at hand. Workflow will be likely tested empirically (pick the workflow that leads to the best performing algo) – pincopallino Oct 13 '16 at 13:16
  • I would propose to write your own library for this kind of issues you have. It's really not a lot of work for this use case and it allows everyone to have this same grammar. You can make it dependent on other packages if your colleagues have access to it. – Jan van der Vegt Oct 13 '16 at 13:25

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.