Splitting before tfidf or after?

When should I perform preprocessing and matrix creation of text data in NLP, before or after train_test_split? Below is my sample code where I have done preprocessing and matrix creation (tfidf) before train_test_split. I want to know will there be data leakage?

corpus = []

for i in range(0 ,len(data1)):
review = re.sub('[^a-zA-Z]', ' ', data1['features'][i])
review = review.lower()
review = review.split()
review = [stemmer.stem(j) for j in review if not j in set(stopwords.words('english'))]
review = ' '.join(review)
corpus.append(review)

from sklearn.feature_extraction.text import TfidfVectorizer
cv = TfidfVectorizer(max_features = 6000)
x = cv.fit_transform(corpus).toarray()

from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
y = le.fit_transform(data1['label'])

from sklearn.model_selection import train_test_split
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size = 0.2, random_state = 69,
stratify = y)

spam_model = MultinomialNB().fit(train_x, train_y)
pred = spam_model.predict(test_x)
c_matrix = confusion_matrix(test_y, pred)
acc_score = accuracy_score(test_y, pred)


1 Answer

You should split before tf-idf.

If you learn tf-idf also on the test set you will have data leakage. In example, you won't have out-of-vocabulary words in inference on the test set, what might happen in the "real world".

• Ok how about the preprocessing of text (like lemmatization, removing stopwords, lowercase etc)? Should I do that before or after the split? Aug 21 at 10:52
• I don't see a difference because those are fixed procedures, they are not data based. Aug 21 at 13:34
• Lemmatization is data based Aug 21 at 15:11
• Also can you provide me with an example where Vectorization is done after splitting data? Because eveywhere I looked, they perform Vectorization before splitting. Aug 22 at 15:08
• The lemmatization I know is not learned. Aug 22 at 19:15