1
$\begingroup$

The docs state that token_pattern is only used if analyzer == 'word':

token_pattern : string

    Regular expression denoting what constitutes a “token”, only used if
    analyzer == 'word'. The default regexp selects tokens of 2 or more 
    alphanumeric characters (punctuation is completely ignored and always 
    treated as a token separator).

Below is my desired pipeline:

analyzer = TfidfVectorizer().build_analyzer()
stemmer = SnowballStemmer('english')

def processed_words(doc):
    return (stemmer.stem(w) for w in analyzer(doc))

vec = TfidfVectorizer(analyzer=processed_words, strip_accents='unicode',
            stop_words='english', token_pattern=r'\b[^_\d\W]+\b')

If I include analyzer=processed_words, then I lose the ability to cut out the thousands of features that are numbers, underscores, and whatever other invalid character sequences specified in the regex.

Is there no way to achieve both stemming and token_pattern matching at the same time? Must I loop through all my documents ahead of time and rejoin the split documents after filtering using regex?

How about if I want to apply stemming AND lemmatization using two analyzers (include also WordNetLemmatizer()) -- how does one handle that?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.