Distribution of text data

Question

How can I identify whether the training data and test data came from same distribution or not?

I tried with TFIDF and cosine similarity

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity

train_data=pd.read_csv('train_dataset.csv')
test_data=pd.read_csv('test_dataset.csv')
tfidf1 = TfidfVectorizer().fit_transform(train_data.text)
tfidf2 = TfidfVectorizer().fit_transform(test_data.text)
# compute and print the cosine similarity matrix
cosine_sim = cosine_similarity(tfidf1, tfidf2).flatten()
print(cosine_sim)

By executing, I got this error ValueError: Incompatible dimension for X and Y matrices: X.shape[1] == 5416 while Y.shape[1] == 11658

Answer 1

On test set, only transform:

tfidf1 = TfidfVectorizer().fit_transform(train_data.text)
tfidf2 = TfidfVectorizer().transform(test_data.text)