I have 1-4 gram text data from wikipedia for 14 categories, which I am using for NE classification. I feed named entity from sentence to lucene indexer which searches named entity from these 14 categories. Issue I am facing is, for single entity I get multiple classes as a result with same score. like while search titanic, indexer gives this result

Score - 11.23 Title - titanic Category - Book

Score - 11.23 Title - titanic Category - Movie

Score - 11.23 Title - titanic Category - Product

now problem is which class to be considered?

I already tried with classifiers (NB,ME in nltk,scikit learn), but as it consider each entity from dataset as feature, it works as indexer only.

Why lucene?

enter image description here


1 Answer 1


I'm not sure I fully understand your question, but it seems to me that you're trying to determine the category of the string/entity "titanic" out of context. Your data tells you that "titanic" could be a book, a movie, or a product, and you want to figure out which one is correct -- is that what you're trying to do?

If so, the problem is that you've dropped the context in which the string/entity "titanic" appears in your original text. For example...

  • In the sentence "I couldn't stop reading Titanic," the word "titanic" refers to a book.
  • In the sentence "Titanic was one of the highest-grossing films of all time," the word "titanic" refers to a movie.
  • In the sentence "The Titanic was the world's largest ocean liner," the word "titanic" refers to a product.

Without that context, there's no way to know which is the correct category. I'd suggest looking into how named entity recognition tools like Stanford NER work -- that will help you better understand how to do something like this. You'll see that the input to an NER tool generally needs to be a sentence, in order to take advantage of the context to properly categorize the extracted entities.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.