Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I know there are questions on how to deal with spelling error NLP - but the question and solution are mainly focused on English where there are tons of library for spell-correction.
Here I am curious what strategy are recommended to build a classifier which is more robust against spelling mistakes. I wonder if it makes sense to make the classifier with character level tokenizer ? or perhaps, upsampling correct data to be misspeled ?
If the language doesn't have any spelling correction tool and you want to take care of spelling errors, practically you'd have to build one by using string similarity measures, and using frequency as an indication of the correct spelling.
In my opinion, very often it's not worth the effort and there's a risk that it would introduce new errors. Statistically, if a spelling error is frequent then it's better to train the model with it so it recognizes it when the model is applied (i.e. exactly as if the spelling was correct), and if the spelling error is rare then it's just noise: it's impossible to get rid of all the noise and it shouldn't affect performance significantly.
