I wonder on which texts should TfidfVectorizer be fitted when using TF-IDF cosine for text similarity. Should TfidfVectorizer be fitted on the texts that are analyzed for text similarity, or some other texts (if so, which one)?

I follow ogrisel's code to compute text similarity via TF-IDF cosine, which fits the TfidfVectorizer on the texts that are analyzed for text similarity (fetch_20newsgroups() in that example):

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.datasets import fetch_20newsgroups
twenty = fetch_20newsgroups()
tfidf = TfidfVectorizer().fit_transform(twenty.data)
from sklearn.metrics.pairwise import linear_kernel
cosine_similarities = linear_kernel(tfidf[0], tfidf[1]).flatten()
print(cosine_similarities) # print TF-IDF cosine similarity between text 1 and 2.
  • $\begingroup$ I am still interested in this question. $\endgroup$ Commented Dec 27, 2023 at 4:48
  • $\begingroup$ Any idea is welcome. $\endgroup$ Commented Dec 27, 2023 at 4:48


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.