I'm trying to get list of n-gram tokens for text

Ex: 'How to use build_analyzer in sklearn feature extraction '

output :['How', 'use', 'build_analyzer', 'sklearn', 'feature', 'extraction', 'How use', 'use build_analyzer', 'build_analzer sklearn', 'sklearn feature', 'feature extraction']

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer

vectorizer = TfidfVectorizer(stop_words = 'english',ngram_range=(1, 2), token_pattern=r'\b\w+\b', min_df=1)

 df['Text'].apply(lambda x : vectorizer.build_analyzer(x))

TypeError: build_analyzer() takes 1 positional argument but 2 were given


build_analyzer() returns a callable that let's you extract the tokenizing step from the transformation pipeline wrapped in the CountVectorizer or TfidfVectorizer. You can do something like this:

analyze = vectorizer.build_analyzer()
df['Text'].apply(lambda x: analyze(x)) #or df['Text'].apply(analyze)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.