# Scikitlearn - TfidfVectorizer - how to use a custom analyzer AND still use token_pattern

The docs state that token_pattern is only used if analyzer == 'word':

token_pattern : string

Regular expression denoting what constitutes a “token”, only used if
analyzer == 'word'. The default regexp selects tokens of 2 or more
alphanumeric characters (punctuation is completely ignored and always
treated as a token separator).


Below is my desired pipeline:

analyzer = TfidfVectorizer().build_analyzer()
stemmer = SnowballStemmer('english')

def processed_words(doc):
return (stemmer.stem(w) for w in analyzer(doc))

vec = TfidfVectorizer(analyzer=processed_words, strip_accents='unicode',
stop_words='english', token_pattern=r'\b[^_\d\W]+\b')


If I include analyzer=processed_words, then I lose the ability to cut out the thousands of features that are numbers, underscores, and whatever other invalid character sequences specified in the regex.

Is there no way to achieve both stemming and token_pattern matching at the same time? Must I loop through all my documents ahead of time and rejoin the split documents after filtering using regex?

How about if I want to apply stemming AND lemmatization using two analyzers (include also WordNetLemmatizer()) -- how does one handle that?