What the difference between these two data formats?

For example, for the Named Entity Recognition task, I learned that index and BIO Encoding are popular data formats to train.

Are they have different features for machine learning, and should I choose input data format following training models' requirements?

# index representation
    "entities": [
            "name": "John J. Smith ",
            "span": [4,8],
            "type": "PERSON"

# BIO Encoding
Yesterday   O   O   O   BOS_O
afternoon   0   O   O   O
,   0   O   O   O_PER
John    I_PER   B_PER   B_PER   B_PER
.   I_PER   I_PER   M_PER   M_PER
Smith   I_PER   I_PER   E_PER   E_PER
traveled    O   0   O   PER_O
to  O   O   O   O_LOC
.   O   O

The BIO format (and its variants) is a standard format for training a sequence labeling model, in particular a Named Entity Recognition (NER) model.

Sequence labeling consists in assigning a label to every token in the sequence, so at the "low level" stages of training and predicting the system must deal with the token and its label, as well as (possibly) other features associated with the token. There are several possible choices to represent an entity through the labels: there must be at least two obviously, and it has been proved that adding at least a special B for the first token in the entity is beneficial.

A json-like format like the one you present can be used as a simplified output of a NER system, typically for applications which only need a list of recognized entities with their type. It's usually more convenient to manipulate but it cannot be used directly by the NER system: it doesn't even contain the full text, it's not tokenized and it doesn't have a label for each token. But assuming that the full text is also provided, this format could also be converted to BIO or some variant but it's more work.

  • If the goal is to provide a dataset which can be fed immediately to train a NER model, then the BIO format is clearly more suitable.
  • If the goal is to provide a convenient format for other usages then something like this JSON format is fine, it's not really a matter of NER.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.