I wonder on which texts should TfidfVectorizer be fitted when using TF-IDF cosine for text similarity. Should TfidfVectorizer be fitted on the texts that are analyzed for text similarity, or some other texts (if so, which one)?
I follow ogrisel's code to compute text similarity via TF-IDF cosine, which fits the TfidfVectorizer on the texts that are analyzed for text similarity (fetch_20newsgroups() in that example):
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.datasets import fetch_20newsgroups
twenty = fetch_20newsgroups()
tfidf = TfidfVectorizer().fit_transform(twenty.data)
from sklearn.metrics.pairwise import linear_kernel
cosine_similarities = linear_kernel(tfidf[0], tfidf[1]).flatten()
print(cosine_similarities) # print TF-IDF cosine similarity between text 1 and 2.
