Scikitlearn - TfidfVectorizer - how to use a custom analyzer AND still use token_pattern

Question

The docs state that token_pattern is only used if analyzer == 'word':

token_pattern : string

    Regular expression denoting what constitutes a “token”, only used if
    analyzer == 'word'. The default regexp selects tokens of 2 or more 
    alphanumeric characters (punctuation is completely ignored and always 
    treated as a token separator).

Below is my desired pipeline:

analyzer = TfidfVectorizer().build_analyzer()
stemmer = SnowballStemmer('english')

def processed_words(doc):
    return (stemmer.stem(w) for w in analyzer(doc))

vec = TfidfVectorizer(analyzer=processed_words, strip_accents='unicode',
            stop_words='english', token_pattern=r'\b[^_\d\W]+\b')

If I include analyzer=processed_words, then I lose the ability to cut out the thousands of features that are numbers, underscores, and whatever other invalid character sequences specified in the regex.

Is there no way to achieve both stemming and token_pattern matching at the same time? Must I loop through all my documents ahead of time and rejoin the split documents after filtering using regex?

How about if I want to apply stemming AND lemmatization using two analyzers (include also WordNetLemmatizer()) -- how does one handle that?

Answer 1

That might not be possible within scikit-learn. Scikit-learn is not designed for extensive text processing.

It might make more sense to define a data processing pipeline outside of scikit-learn. Then pass the outputs to a simplified version of TfidfVectorizer().