How can I identify whether the training data and test data came from same distribution or not?

I tried with TFIDF and cosine similarity

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity

tfidf1 = TfidfVectorizer().fit_transform(train_data.text)
tfidf2 = TfidfVectorizer().fit_transform(test_data.text)
# compute and print the cosine similarity matrix
cosine_sim = cosine_similarity(tfidf1, tfidf2).flatten()

By executing, I got this error ValueError: Incompatible dimension for X and Y matrices: X.shape[1] == 5416 while Y.shape[1] == 11658


1 Answer 1


On test set, only transform:

tfidf1 = TfidfVectorizer().fit_transform(train_data.text)
tfidf2 = TfidfVectorizer().transform(test_data.text)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.