I am trying to recognize and classify the entity types based on the IOB/Sequence labeling.

I was able to use nltk.ne_chunk() which is already trained to recognize named entities using their train set.

I was wondering if there is any way to train the algorithm using my on training set and labels using training data like the following,

send        O
sms         B-TASK 
8714349616  B-MOB
how         B-MSG
are         I-MSG
you         I-MSG

sms        B-TASK        
how        B-MSG
are        I-MSG
you        I-MSG        
to         O
8714349616 B-MOB

is it possible using NLTK ? Any examples or Tutorials ?


1 Answer 1


Have a look on conditional random field. Conditional random fields are a probabilistic framework for labeling and segmenting structured data, such as sequences, trees and lattices. The underlying idea is that of defining a conditional probability distribution over label sequences given a particular observation sequence, rather than a joint distribution over both label and observation sequences. The primary advantage of CRFs is their conditional nature, resulting in the relaxation of the independence assumptions. Additionally, CRFs avoid the label bias problem. It is often used for labeling or parsing of sequential data. CRF++ is a simple, customizable, and open source implementation of Conditional Random Fields (CRFs)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.