Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I've just come across chunking and I can't get my head around why is it necessary? I know that it is used for 'named entity recognition'. I have few questions:
BIO(L) tagging is important (but as you correctly noted, not necessary) part of a NER pipeline. Main idea behind such split is to facilitate learning in following manner.
Take English as an example, some words will (most likely) never end a Named Entity, like adjectives, so the model will never tag them as the L(ast) part of a named entity. The same applies to the L-tag.
What is crucial, is that many models, like Conditional Random Fields, learn not only tags themselves, but also the transition probability, so, if you will get some chunk of text that is tagged as B_ O_ L_, that sequence is incorrect, but when you learn the transitions as well, model will figure out, that if you get a strong Beginning and End, the inside part should also be a entity part.
