What is the best practice to classify category of named entity in sentence

Question

I have 1-4 gram text data from wikipedia for 14 categories, which I am using for NE classification. I feed named entity from sentence to lucene indexer which searches named entity from these 14 categories. Issue I am facing is, for single entity I get multiple classes as a result with same score. like while search titanic, indexer gives this result

Score - 11.23 Title - titanic Category - Book

Score - 11.23 Title - titanic Category - Movie

Score - 11.23 Title - titanic Category - Product

now problem is which class to be considered?

I already tried with classifiers (NB,ME in nltk,scikit learn), but as it consider each entity from dataset as feature, it works as indexer only.

Why lucene?

Answer 1

I'm not sure I fully understand your question, but it seems to me that you're trying to determine the category of the string/entity "titanic" out of context. Your data tells you that "titanic" could be a book, a movie, or a product, and you want to figure out which one is correct -- is that what you're trying to do?

If so, the problem is that you've dropped the context in which the string/entity "titanic" appears in your original text. For example...

• In the sentence "I couldn't stop reading Titanic," the word "titanic" refers to a book.
• In the sentence "Titanic was one of the highest-grossing films of all time," the word "titanic" refers to a movie.
• In the sentence "The Titanic was the world's largest ocean liner," the word "titanic" refers to a product.

Without that context, there's no way to know which is the correct category. I'd suggest looking into how named entity recognition tools like Stanford NER work -- that will help you better understand how to do something like this. You'll see that the input to an NER tool generally needs to be a sentence, in order to take advantage of the context to properly categorize the extracted entities.