I'm trying to implement NER using BIO annotation. For example
"I went to the United States" [O, O, O, B, I, I]
where B and I denote the beginning and 'I' the following of the entity.
However, when I use a vanilla BERT to do classification(whether it belongs it 'B', 'I', 'O') at each position of the sequence, I encounter cases where 'O' is followed by an 'I'. There are no cases in the data that exhibit ('O', 'I') pattern since there's always a 'B' or 'I' in front. Obviously, there's nothing to enforce the model to exclude such a pattern but I would like to somehow incorporate it into the model(like a transitioning probability from 'O' to 'I' being 0 or something).
I took a look at conditional random fields on top of BERT that is trying to do something very similar but the prediction somehow still contained these 'O' 'I' patterns.