The docs state that
token_pattern is only used if analyzer == 'word':
token_pattern : string Regular expression denoting what constitutes a “token”, only used if analyzer == 'word'. The default regexp selects tokens of 2 or more alphanumeric characters (punctuation is completely ignored and always treated as a token separator).
Below is my desired pipeline:
analyzer = TfidfVectorizer().build_analyzer() stemmer = SnowballStemmer('english') def processed_words(doc): return (stemmer.stem(w) for w in analyzer(doc)) vec = TfidfVectorizer(analyzer=processed_words, strip_accents='unicode', stop_words='english', token_pattern=r'\b[^_\d\W]+\b')
If I include
analyzer=processed_words, then I lose the ability to cut out the thousands of features that are numbers, underscores, and whatever other invalid character sequences specified in the regex.
Is there no way to achieve both stemming and token_pattern matching at the same time? Must I loop through all my documents ahead of time and rejoin the split documents after filtering using regex?
How about if I want to apply stemming AND lemmatization using two analyzers (include also
WordNetLemmatizer()) -- how does one handle that?