1
$\begingroup$
class sklearn.feature_extraction.text.TfidfVectorizer(
*, input='content', encoding='utf-8', decode_error='strict', strip_accents=None, 
lowercase=True, preprocessor=None, tokenizer=None, analyzer='word', 
stop_words=None, token_pattern='(?u)\b\w\w+\b', **ngram_range=(1, 1)**, 
**max_df=1.0**, **min_df=1**, max_features=None, vocabulary=None, binary=False, 
dtype=<class 'numpy.float64'>, norm='l2', use_idf=True, smooth_idf=True, 
sublinear_tf=False)

These are the hyperparameters of sklearn's TF-IDF vectorizer.

The ngram_range parameter takes in the possible range of ngrams to extract from the given documents.

I couldn't help but wonder how does the vectorizer come up with the possible ngrams (say bigrams or trigrams) for the documents supplied as word vector features. Does it use only min_df and max_df to come up with them or is there something deeper than what meets the eye going on underneath?

$\endgroup$
2
  • $\begingroup$ It has other parameters too i.e. stop_words, max_features, etc. What is your doubt? Have you tried building from a small doc and it is not working as expected? $\endgroup$
    – 10xAI
    May 21 at 10:29
  • $\begingroup$ @10xAI I recently used the Tf-Idf vectorizer on a document containing about 2.2k news articles. I changed the min_df parameter of the vectorizer from 8 to 3 and retained the max_df parameter's value at 895, the number of bigrams extracted increased from 6 to 8. My doubt lies in how the vectorizer extracts and balances the no of unigrams and bigrams used in the final feature set. $\endgroup$ May 22 at 4:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.