1
$\begingroup$

I'm new to NER and I've been trying to extract names using Spacy. Here's my code:

import spacy
spacy_nlp = spacy.load('en_core_web_sm')

doc = spacy_nlp(text.strip())

# create sets to hold words 
named_entities = set()
money_entities = set()
organization_entities = set()
location_entities = set()
time_indicator_entities = set()

for i in doc.ents:
    entry = str(i.lemma_).lower()
    text = text.replace(str(i).lower(), "")
    # Time indicator entities detection
    if i.label_ in ["TIM", "DATE"]:
        time_indicator_entities.add(entry)
    # money value entities detection
    elif i.label_ in ["MONEY"]:
        money_entities.add(entry)
    # organization entities detection
    elif i.label_ in ["ORG"]:
        organization_entities.add(entry)
    # Geographical and Geographical entities detection
    elif i.label_ in ["GPE", "GEO"]:
        location_entities.add(entry)
    # extract artifacts, events and natural phenomenon from text
    elif i.label_ in ["ART", "EVE", "NAT", "PERSON"]:
        named_entities.add(entry.title())

The model seems to have a decent accuracy with certain kinds of names. However it is unaware of how people’s names can differ around the world (not adapted to suit cultural differences). Is there a possible workaround to avoid this bias?

$\endgroup$
1
$\begingroup$

The NER model performance on a particular text depends on which data it was trained with originally, and naturally the standard models (like en_core_web_sm) are trained with English data which doesn't contain a lot of names from non-US/UK origin (same for other kinds of entities like organizations or locations).

Better performance can be achieved by training your own model with your own labelled data, but that requires you (or somebody) to annotate a reasonably large sample of data manually.

$\endgroup$
2
  • $\begingroup$ How much data will be required for NER training purpose. I have around 15 entities to add and is it possible to give a count on the quantity of data required ? $\endgroup$
    – Aniiya0978
    Oct 12 at 5:13
  • $\begingroup$ @Aniiya0978 be careful that NER is not about the number of entities, it's about training a model which can recognize any entity based on the sentence context, as I was explaining in this answer recently. If you have a fixed list of entities, NER is not the right approach. In general NER models are trained with at least thousands of annotated sentences. $\endgroup$
    – Erwan
    Oct 12 at 11:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.