I am trying to make a custom entity model for an NER application using spacy. In several NLP projects, I have converted all the data to lowercase and applied several ML techniques. For NER also should I have to convert the data to lowercase. Or why it is necessary to convert to lower case. Is it a mandate one which will affect the accuracy of the model adversely if not converted to lowercase.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Converting to lower case is a historical method to combat data sparsity. The idea is that if you don't have a lot of data, case usually does't matter, so remove the meaningless variable.
But for NER case is an important clue - capital words are more likely to be proper nouns, for example. So you definitely don't want to lower case things.
In general, aggressive preprocessing was necessary in the past, when data was scarce and it was hard to fit everything in memory if you had a lot of data. But that's no longer the case, so it's better to use unmodified text to give the models as many clues as possible.
