12
$\begingroup$

I have been working in NLTK for a while using Python. The problem I am facing is that their is no help available on training NER in NLTK with my custom data. They have used MaxEnt and trained it on ACE corpus. I have searched on the web a lot but I could not find any way that can be used to train NLTK's NER.

If anyone can provide me with any link/article/blog etc which can direct me to Training Datasets Format used in training NLTK's NER so I can prepare my Datasets on that particular format. And if I am directed to any link/article/blog etc which can help me TRAIN NLTK's NER for my own data.

This is a question widely searched and least answered. Might be helpful for someone in the future whose working with NER.

Improve this question
$\endgroup$
1

3 Answers 3

Reset to default
4
$\begingroup$

Training a model, related to information extraction, in general, and named entity recognition/resolution (NER), in particular, is described in detail in Chapter 7 of the NLTK Book, available online at this URL: http://www.nltk.org/book/ch07.html.

Additionally, I think that you might find useful my related answer on Cross Validated site. It has a lot of references to relevant sources on NER and related topics as well as to various related software tools.

Improve this answer
$\endgroup$
4
  • $\begingroup$ They do not mention how to train the NER model on custom data, can you tell how to do it? $\endgroup$
    – Hima Varsha
    Commented Dec 28, 2016 at 12:29
  • 1
    $\begingroup$ @HimaVarsha I'm not an expert in this area. However, ... I think that NLTK NER model comes pre-trained on the conll2000 corpus, hence no info in NLTK book. Check the following resources: 1. nltk-trainer.readthedocs.io (most likely what you need; probably the Training IOB Chunkers section). 2. sujitpal.blogspot.com/2012/11/… (might be useful as well). 3. nlp.stanford.edu/software/crf-faq.shtml#a (in case you use or will decide to use Stanford NER software). $\endgroup$
    – Aleksandr Blekh
    Commented Dec 28, 2016 at 21:22
  • $\begingroup$ I think stanfordcrf implementation does take custom data, but NTLK NER comes just pre-trained. The Training IOB Chunkers is just chunking right? Or does it even do NER? $\endgroup$
    – Hima Varsha
    Commented Dec 29, 2016 at 5:47
  • $\begingroup$ @HimaVarsha Please pay more attention to advice you're getting. If you'd read post via link #2 above carefully, you'd see that the code there does both NER model training and running. I don't think I can help you beyond the advice above. $\endgroup$
    – Aleksandr Blekh
    Commented Dec 29, 2016 at 8:37
3
$\begingroup$

Is this article good enough? http://www.succeed-project.eu/wiki/index.php/NLTK#Input_format_for_training

There is explanation about how corpus should look like.

Your data needs to be in IOB format (word tag chunktag) to make it work.
Eric NNP B-PERSON
is VB O
the AT B-NP
CEO NN I-NP
of IN O
Google NNP B-ORGANIZATION

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ It would be ideal to post a short summary of the article in this answer. $\endgroup$
    – sheldonkreger
    Commented Mar 2, 2015 at 23:53
1
$\begingroup$

I found this tutorial quite helpful: Complete guide to build your own Named Entity Recognizer with Python He uses the Groningen Meaning Bank (GMB) corpus to train his NER chunk.

After that you can check this tutorial from the same person: Training a NER System Using a Large Dataset Where he uses scikit learn to improve the performance of his system.

Finally some really useful tutorials can be found here: NLTK tutorial This guy has a youtube channel with a lot of tutorial in many subjects (ML,NLP, Python...)

Hope it helps.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.