0
$\begingroup$

I'm using gensim in Python for topic modeling. Currently, I have one problem. If I don't lemmatize, human names will appear as 'Most Relevant Terms for Topic,' but after lemmatization, the human names disappear from 'Most Relevant Terms for Topic.'

Could you please recommend another lemmatization code that can handle the human name issue? Or, is there literature reporting that lemmatization may not properly handle human names? I used the code below.

Thank you!

def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV']):
    """https://spacy.io/api/annotation"""
    texts_out = []
    for sent in texts:
        doc = nlp(" ".join(sent)) 
        texts_out.append([token.lemma_ for token in doc if token.pos_ in allowed_postags])
    return texts_out

import spacy
# Remove Stop Words
data_words_nostops = remove_stopwords(data_words)
# Form Bigrams
data_words_bigrams = make_bigrams(data_words_nostops)
# Initialize spacy 'en' model, keeping only tagger component (for efficiency)
nlp = spacy.load("en_core_web_sm", disable=['parser', 'ner'])
# Do lemmatization keeping only noun, adj, vb, adv
data_lemmatized = lemmatization(data_words_bigrams, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV'])
print(data_lemmatized[:1])

import gensim.corpora as corpora
# Create Dictionary
id2word = corpora.Dictionary(data_lemmatized)
# Create Corpus
texts = data_lemmatized
# Term Document Frequency
corpus = [id2word.doc2bow(text) for text in texts]
# View
print(corpus[:1])
$\endgroup$
2
  • $\begingroup$ Use a named-entity-recognition model (NER) to recognize names (spacy has one by default) and use the model to replace the names with a <NAME> tag. $\endgroup$ Commented Aug 6, 2022 at 19:35
  • $\begingroup$ Thank you for your comment. In my code, 'ner' is marked as 'disable.' But even if I remove that code, the names still disappear. If possible, could you please give me some example code? $\endgroup$
    – SEan1820
    Commented Aug 7, 2022 at 22:43

1 Answer 1

2
$\begingroup$

Names qualify as a 'PROPN' postag. Adding it to the allowed_postags list should likely fix the issue.

Edit: a simplified example (accepts raw text, provides a list of postags as debug output and returns lemmatized unsplit text). Basically nothing should be changed apart from allowed_postags, but this might give you some insight on the tags you truly need.

import spacy

nlp = spacy.load('en_core_web_sm', disable=['parser', 'ner'])

def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV', 'PROPN']):
    texts_out = []
    for sent in texts:
        doc = nlp(sent)
        print(list(token.pos_ for token in doc))
        texts_out.append(' '.join([token.lemma_ for token in doc if token.pos_ in allowed_postags]))
    return texts_out

texts = [
    'Alex Jones caught in lie about Sandy Hook texts during brutal cross-examination',
    'China sets military drills around Taiwan',
    'Analysis: "Slap in the face": Biden\'s fist bump with MBS fails to pay off',
    'Closing arguments expected in WNBA star Brittney Griner\'s Russia drug-smuggling trial'
]

lemmatization(texts)
$\endgroup$
1
  • $\begingroup$ Thank you for your answer. If possible, can you give me some example code? $\endgroup$
    – SEan1820
    Commented Aug 7, 2022 at 22:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.