# How to choose and create natural language data for machine learning

What the difference between these two data formats?

For example, for the Named Entity Recognition task, I learned that index and BIO Encoding are popular data formats to train.

Are they have different features for machine learning, and should I choose input data format following training models' requirements?

# index representation
"entities": [
{
"name": "John J. Smith ",
"span": [4,8],
"type": "PERSON"
}

# BIO Encoding
Tokens  IO  BIO BMEWO   BMEWO+
Yesterday   O   O   O   BOS_O
afternoon   0   O   O   O
,   0   O   O   O_PER
John    I_PER   B_PER   B_PER   B_PER
J   I_PER   I_PER   M_PER   M_PER
.   I_PER   I_PER   M_PER   M_PER
Smith   I_PER   I_PER   E_PER   E_PER
traveled    O   0   O   PER_O
to  O   O   O   O_LOC
Washington
I_LOC   B_LOC   W_LOC   W_LOC
.   O   O


Sequence labeling consists in assigning a label to every token in the sequence, so at the "low level" stages of training and predicting the system must deal with the token and its label, as well as (possibly) other features associated with the token. There are several possible choices to represent an entity through the labels: there must be at least two obviously, and it has been proved that adding at least a special B for the first token in the entity is beneficial.