How can I identify whether the training data and test data came from same distribution or not?

I tried with TFIDF and cosine similarity

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity

tfidf1 = TfidfVectorizer().fit_transform(train_data.text)
tfidf2 = TfidfVectorizer().fit_transform(test_data.text)
# compute and print the cosine similarity matrix
cosine_sim = cosine_similarity(tfidf1, tfidf2).flatten()

By executing, I got this error ValueError: Incompatible dimension for X and Y matrices: X.shape[1] == 5416 while Y.shape[1] == 11658


1 Answer 1


On test set, only transform:

tfidf1 = TfidfVectorizer().fit_transform(train_data.text)
tfidf2 = TfidfVectorizer().transform(test_data.text)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.