0
$\begingroup$

When should I perform preprocessing and matrix creation of text data in NLP, before or after train_test_split? Below is my sample code where I have done preprocessing and matrix creation (tfidf) before train_test_split. I want to know will there be data leakage?

corpus = []

for i in range(0 ,len(data1)):
    review = re.sub('[^a-zA-Z]', ' ', data1['features'][i])
    review = review.lower()
    review = review.split()
    review = [stemmer.stem(j) for j in review if not j in set(stopwords.words('english'))]
    review = ' '.join(review)
    corpus.append(review)

from sklearn.feature_extraction.text import TfidfVectorizer
cv = TfidfVectorizer(max_features = 6000)
x = cv.fit_transform(corpus).toarray()

from sklearn.preprocessing import LabelEncoder
le = LabelEncoder()
y = le.fit_transform(data1['label'])

from sklearn.model_selection import train_test_split
train_x, test_x, train_y, test_y = train_test_split(x, y, test_size = 0.2, random_state = 69, 
                                                                                stratify = y)

spam_model = MultinomialNB().fit(train_x, train_y)
pred = spam_model.predict(test_x)
c_matrix = confusion_matrix(test_y, pred)
acc_score = accuracy_score(test_y, pred)
$\endgroup$

1 Answer 1

3
$\begingroup$

You should split before tf-idf.

If you learn tf-idf also on the test set you will have data leakage. In example, you won't have out-of-vocabulary words in inference on the test set, what might happen in the "real world".

$\endgroup$
7
  • $\begingroup$ Ok how about the preprocessing of text (like lemmatization, removing stopwords, lowercase etc)? Should I do that before or after the split? $\endgroup$
    – spectre
    Commented Aug 21, 2021 at 10:52
  • 1
    $\begingroup$ I don't see a difference because those are fixed procedures, they are not data based. $\endgroup$ Commented Aug 21, 2021 at 13:34
  • $\begingroup$ Lemmatization is data based $\endgroup$
    – spectre
    Commented Aug 21, 2021 at 15:11
  • $\begingroup$ Also can you provide me with an example where Vectorization is done after splitting data? Because eveywhere I looked, they perform Vectorization before splitting. $\endgroup$
    – spectre
    Commented Aug 22, 2021 at 15:08
  • $\begingroup$ The lemmatization I know is not learned. $\endgroup$ Commented Aug 22, 2021 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.