Extracting Names using NER | Spacy

Question

I'm new to NER and I've been trying to extract names using Spacy. Here's my code:

import spacy
spacy_nlp = spacy.load('en_core_web_sm')

doc = spacy_nlp(text.strip())

# create sets to hold words 
named_entities = set()
money_entities = set()
organization_entities = set()
location_entities = set()
time_indicator_entities = set()

for i in doc.ents:
    entry = str(i.lemma_).lower()
    text = text.replace(str(i).lower(), "")
    # Time indicator entities detection
    if i.label_ in ["TIM", "DATE"]:
        time_indicator_entities.add(entry)
    # money value entities detection
    elif i.label_ in ["MONEY"]:
        money_entities.add(entry)
    # organization entities detection
    elif i.label_ in ["ORG"]:
        organization_entities.add(entry)
    # Geographical and Geographical entities detection
    elif i.label_ in ["GPE", "GEO"]:
        location_entities.add(entry)
    # extract artifacts, events and natural phenomenon from text
    elif i.label_ in ["ART", "EVE", "NAT", "PERSON"]:
        named_entities.add(entry.title())

The model seems to have a decent accuracy with certain kinds of names. However it is unaware of how people’s names can differ around the world (not adapted to suit cultural differences). Is there a possible workaround to avoid this bias?

Answer 1

The NER model performance on a particular text depends on which data it was trained with originally, and naturally the standard models (like en_core_web_sm) are trained with English data which doesn't contain a lot of names from non-US/UK origin (same for other kinds of entities like organizations or locations).

Better performance can be achieved by training your own model with your own labelled data, but that requires you (or somebody) to annotate a reasonably large sample of data manually.