I'm trying to use MITIE to extract named entities from short text. I'm interested in entities such as dates, times, names, and locations. Out of the box, MITIE only recognises names, locations, and organisations. I'd like to train it to recognise dates, times and other categories as well. From looking at the structure of MITIE's directories and from the dlib website, I gather that this is done via an SVM. Is this correct?

With regards to adding new categories to the named entity recogniser, I have several questions:

  1. Can this be done in an augmentative fashion? That is, given an existing NER system, can I just add categories examples and train it to recognise those as well? Or do I need to train models from scratch?
  2. If I do need train models from scratch, what dataset can I use to do this?
  3. Related to adding new examples, is there an online method that I can use, feeding the system new examples and categories as and when I need to?
  • $\begingroup$ blog.booking.com/named-entity-classification.html This might not answer all of your questions, but definitely worth reading the article mentioned above. $\endgroup$ – rbhat Jul 11 '17 at 8:58
  • $\begingroup$ Thanks @user2997081 - that's where I started my search, and although the article is good, it took me a fair amount of searching to understand MITIE better. I hope my answer can be of help to others searching along similar lines. $\endgroup$ – Navaneethan Santhanam Jul 14 '17 at 6:46

After having used MITIE for a few weeks, I feel like I at least have enough to answer my basic questions:

  1. (and 3.) All models need to be trained from scratch - there is no online method to add new samples to the model as they come in. This is unfortunate because MITIE takes at least 45 minutes to an hour to train on a ~20k-sized dataset.
  2. The datasets I used were ATIS, CoNLL 2003, and DBpedia

I've found MITIE to be quite good as far as classification accuracy goes, although it takes a bit of work to prepare datasets for it.

| improve this answer | |
  • $\begingroup$ Could you please add a small reproducible example on how to use MITIE for NER & POS in Python. Their github is rather unclear and confusing. $\endgroup$ – kRazzy R Oct 4 '17 at 22:50
  • $\begingroup$ @kRazzyR it'll take me at least a few days to put this together. In the meantime, have a look at this example (for NER) - github.com/mit-nlp/MITIE/blob/master/examples/python/…. I found it to be quite straightforward, and it's only a few lines once you've cloned their github repo and downloaded the necessary files (from their README). Let me know if that works for you. If not, I'll put together an example. I've not used MITIE for POS (not sure they even have support for POS), actually. $\endgroup$ – Navaneethan Santhanam Oct 5 '17 at 10:51
  • $\begingroup$ If you want, Spacy is great for dates, names, and somewhat for times. MITIE is better-ish for locations. Addresses are always awkward. Combining the two may work without needing a ton of training data. $\endgroup$ – Andrew Scott Evans Aug 23 '19 at 5:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.