Lost human names after 'Lemmatization' for topic modeling in python

Question

I'm using gensim in Python for topic modeling. Currently, I have one problem. If I don't lemmatize, human names will appear as 'Most Relevant Terms for Topic,' but after lemmatization, the human names disappear from 'Most Relevant Terms for Topic.'

Could you please recommend another lemmatization code that can handle the human name issue? Or, is there literature reporting that lemmatization may not properly handle human names? I used the code below.

Thank you!

def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV']):
    """https://spacy.io/api/annotation"""
    texts_out = []
    for sent in texts:
        doc = nlp(" ".join(sent)) 
        texts_out.append([token.lemma_ for token in doc if token.pos_ in allowed_postags])
    return texts_out

import spacy
# Remove Stop Words
data_words_nostops = remove_stopwords(data_words)
# Form Bigrams
data_words_bigrams = make_bigrams(data_words_nostops)
# Initialize spacy 'en' model, keeping only tagger component (for efficiency)
nlp = spacy.load("en_core_web_sm", disable=['parser', 'ner'])
# Do lemmatization keeping only noun, adj, vb, adv
data_lemmatized = lemmatization(data_words_bigrams, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV'])
print(data_lemmatized[:1])

import gensim.corpora as corpora
# Create Dictionary
id2word = corpora.Dictionary(data_lemmatized)
# Create Corpus
texts = data_lemmatized
# Term Document Frequency
corpus = [id2word.doc2bow(text) for text in texts]
# View
print(corpus[:1])

Answer 1

Names qualify as a 'PROPN' postag. Adding it to the allowed_postags list should likely fix the issue.

Edit: a simplified example (accepts raw text, provides a list of postags as debug output and returns lemmatized unsplit text). Basically nothing should be changed apart from allowed_postags, but this might give you some insight on the tags you truly need.

import spacy

nlp = spacy.load('en_core_web_sm', disable=['parser', 'ner'])

def lemmatization(texts, allowed_postags=['NOUN', 'ADJ', 'VERB', 'ADV', 'PROPN']):
    texts_out = []
    for sent in texts:
        doc = nlp(sent)
        print(list(token.pos_ for token in doc))
        texts_out.append(' '.join([token.lemma_ for token in doc if token.pos_ in allowed_postags]))
    return texts_out

texts = [
    'Alex Jones caught in lie about Sandy Hook texts during brutal cross-examination',
    'China sets military drills around Taiwan',
    'Analysis: "Slap in the face": Biden\'s fist bump with MBS fails to pay off',
    'Closing arguments expected in WNBA star Brittney Griner\'s Russia drug-smuggling trial'
]

lemmatization(texts)