2
$\begingroup$

I'm new to nlp. Recently I wanted to do little nlp tasks, and faced strange thing. That is I have run the following code

from sklearn.feature_extraction.text import TfidfVectorizer

docs = ["strange event"]
tfIdf_vectorizer = TfidfVectorizer(analyzer='word', tokenizer=word_tokenize,
                                   stop_words=stopwords, ngram_range=(1, 2), use_idf=True,
                                   norm='l2')
tfidf = tfIdf_vectorizer.fit_transform(docs)
print(tfidf)

and see the following result

  (0, 2)    0.5773502691896258
  (0, 0)    0.5773502691896258
  (0, 1)    0.5773502691896258

shouldn't the tfidf of one single document be zero? (Since the IDF=log(1/1)=0)

$\endgroup$

1 Answer 1

5
$\begingroup$

It is because, by default sklearn's TF-IDF vectorizer will normalize the results. See the the Tf-IDF Term Weighting section of the User Guide. For your example,

n = 1
tf = 3
df = 1
idf = np.log(n/df)+1 = 1

You have 3 terms with identical frequency. So, the L2 normalized tf-idf is computed as

abs(1)/sqrt(1+1+1) = 0.577
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.