1
$\begingroup$

I know there are questions on how to deal with spelling error NLP - but the question and solution are mainly focused on English where there are tons of library for spell-correction.

Here I am curious what strategy are recommended to build a classifier which is more robust against spelling mistakes. I wonder if it makes sense to make the classifier with character level tokenizer ? or perhaps, upsampling correct data to be misspeled ?

$\endgroup$
1

1 Answer 1

1
$\begingroup$

If the language doesn't have any spelling correction tool and you want to take care of spelling errors, practically you'd have to build one by using string similarity measures, and using frequency as an indication of the correct spelling.

In my opinion, very often it's not worth the effort and there's a risk that it would introduce new errors. Statistically, if a spelling error is frequent then it's better to train the model with it so it recognizes it when the model is applied (i.e. exactly as if the spelling was correct), and if the spelling error is rare then it's just noise: it's impossible to get rid of all the noise and it shouldn't affect performance significantly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.