0
$\begingroup$

Will the spacy V3 model get affected by imbalanced entities? I have got a dataset annotated in spacy format and if I look into my custom entities the rations are different for different entities. For example, one entity say 'flex' is more than 2500 but I also have an entity say 'door' which is just 21. I trained my spacy model and evaluated using spacy.evaluate(examples). I'm getting f1-score of 0.64, precision of 1.0 and recall of 0.47. I want to know whether this entity imbalance is affecting model performance?. If yes is there a way to solve this issue? Any help on this will be greatly appreciated.

$\endgroup$

1 Answer 1

3
$\begingroup$

The imbalance between entities is unavoidable: some entities are naturally more frequent than others. It would likely cause various other biases to try to oversample real text in order to increase the number of occurrences.

The imbalance does affect performance of course, it's easier to correctly recognize a frequent entity like "New York" than for instance "Cork". But this is the statistical game, there are always going to be some errors somewhere.

Finally it's important to keep in mind that NER is not primarily meant to recognize only a finite set of predefined entities seen in the training data. On the contrary, the goal is to use clues in the text in order to capture any entity present in the text, whether it has been seen in the training set or not. So in theory the training data should provide a representative sample of the different contexts in which NEs can appear, and this should be sufficient to recognize any entity independently of their frequency. But in reality it's practically impossible to have a perfect representative sample of the contexts, of course.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.