# How does sklearn's tf-idf vectorizer pick the bigrams and trigrams?

class sklearn.feature_extraction.text.TfidfVectorizer(
*, input='content', encoding='utf-8', decode_error='strict', strip_accents=None,
lowercase=True, preprocessor=None, tokenizer=None, analyzer='word',
stop_words=None, token_pattern='(?u)\b\w\w+\b', **ngram_range=(1, 1)**,
**max_df=1.0**, **min_df=1**, max_features=None, vocabulary=None, binary=False,
dtype=<class 'numpy.float64'>, norm='l2', use_idf=True, smooth_idf=True,
sublinear_tf=False)


These are the hyperparameters of sklearn's TF-IDF vectorizer.

The ngram_range parameter takes in the possible range of ngrams to extract from the given documents.

I couldn't help but wonder how does the vectorizer come up with the possible ngrams (say bigrams or trigrams) for the documents supplied as word vector features. Does it use only min_df and max_df to come up with them or is there something deeper than what meets the eye going on underneath?

• It has other parameters too i.e. stop_words, max_features, etc. What is your doubt? Have you tried building from a small doc and it is not working as expected? May 21 at 10:29
• @10xAI I recently used the Tf-Idf vectorizer on a document containing about 2.2k news articles. I changed the min_df parameter of the vectorizer from 8 to 3 and retained the max_df parameter's value at 895, the number of bigrams extracted increased from 6 to 8. My doubt lies in how the vectorizer extracts and balances the no of unigrams and bigrams used in the final feature set. May 22 at 4:46