3
$\begingroup$

I'm working a multi-class text classification project.

After splitting the dataset into train and test datasets, I've applied the below function on the train dataset (AKA pre processing):

STOPWORDS = set(stopwords.words('english'))

def clean_text(text):   
    # lowercase text
    text = text.lower() 
    
    # delete bad symbols
    text = re.sub(r"(@\[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)|^rt|http.+?", "", text)  
  
    # delete stopwords from text
    text = ' '.join(word for word in text.split() if word not in STOPWORDS) 

    # Stemming the words
    text = ' '.join([stemmer.stem(word) for word in text.split()])
    
    return text

To my surprise, I've got much worst results (i.e. va_accuracy) applying on the train dataset rather than just "do nothing" (59% vs 69%)

I've literally commented out the apply line in the below section:

all_data = dataset.sample(frac=1).reset_index(drop=True)
train_df, valid = train_test_split(all_data, test_size=0.2)

train_df['text'] = train_df['text'].apply(clean_text)

What am I missing? How can it be that pre processing steps decreased accuracy?

A bit more info

I forgot to mention I'm using the below to tokenize the text:

X_train = train.iloc[:, :-1]
y_train = train.iloc[:, -1:]
X_test = valid.iloc[:, :-1]
y_test = valid.iloc[:, -1:]

weights = class_weight.compute_class_weight(class_weight='balanced', classes=np.unique(y_train), 
                                            y=y_train.values.reshape(-1))
le = LabelEncoder()
le.fit(weights)
class_weights_dict = dict(zip(le.transform(list(le.classes_)), weights))


tokenizer = Tokenizer(num_words=vocab_size, oov_token='<OOV>')
tokenizer.fit_on_texts(X_train['text'])

train_seq = tokenizer.texts_to_sequences(X_train['text'])
train_padded = pad_sequences(train_seq, maxlen=max_length, padding=padding_type, truncating=trunc_type)

validation_seq = tokenizer.texts_to_sequences(X_test['text'])
validation_padded = pad_sequences(validation_seq, maxlen=max_length, padding=padding_type, truncating=trunc_type)

Later on I'm fitting all into the model as follows:

model = Sequential()
model.add(Embedding(vocab_size, embedding_dim, input_length=train_padded.shape[1]))

model.add(Conv1D(48, len(GROUPS), activation='relu', padding='valid'))
model.add(GlobalMaxPooling1D())
model.add(Dropout(0.5))

model.add(Flatten())
model.add(Dropout(0.5))

model.add(Dense(len(GROUPS), activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

epochs = 100
batch_size = 32

history = model.fit(train_padded, training_labels, shuffle=True ,
                    epochs=epochs, batch_size=batch_size,
                    class_weight=class_weights_dict,
                    validation_data=(validation_padded, validation_labels),
                    callbacks=[ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=5, min_lr=0.0001), 
                               EarlyStopping(monitor='val_loss', mode='min', patience=2, verbose=1),
                               EarlyStopping(monitor='val_accuracy', mode='max', patience=5, verbose=1)])
$\endgroup$
2
  • 1
    $\begingroup$ What model type did you use and what's the domain of the task? $\endgroup$
    – Jonathan
    Oct 6, 2022 at 13:31
  • $\begingroup$ @Jonathan Not sure what you're asking. The domain of the task is multi class text classification. $\endgroup$
    – Ben
    Oct 6, 2022 at 13:37

4 Answers 4

9
$\begingroup$

You have to apply the same preprocessing to the test data.

Based on your code you apply the clean_text function only to train data but then predict on test/validation data that was not cleaned. That means that your model learns on clean data but you want it to predict on raw data which contains things the model never seen (because it was removed from the train dataset) which will result in worse performance.

Edit after discussion in comments:

You can either preprocess all data at the same time before splitting and have

all_data = dataset.sample(frac=1).reset_index(drop=True)
all_data['text'] = all_data['text'].apply(clean_text)
train_df, valid = train_test_split(all_data, test_size=0.2)

or just apply the sample preprocessing to the valid dataset after you split the data

all_data = dataset.sample(frac=1).reset_index(drop=True)
train_df, valid = train_test_split(all_data, test_size=0.2)
train_df['text'] = train_df['text'].apply(clean_text)
valid ['text'] = valid ['text'].apply(clean_text)

It is important to apply the same preprocessing for all data that goes into the model so all input is in the same format. This means that when you deploy your model into application you will also have to apply the same preprocessing to any text input coming into the model before making prediction.

$\endgroup$
6
  • $\begingroup$ You're correct, I do not apply the clean_text method on the test data. In which stage should I perform it? From what I understood pre-processing should be applied on the train data only. $\endgroup$
    – Ben
    Oct 7, 2022 at 13:50
  • $\begingroup$ Preprocessing should be applied to both so you have all data in the same format. In your case during training and validation you can just do valid['text'] = valid['text'].apply(clean_text) where you preprocess the train_df. When you deploy your application and have real input coming you should preprocess the input before you tokenize it and feed it into your model. $\endgroup$
    – aEmQy01b
    Oct 7, 2022 at 13:56
  • $\begingroup$ Can't I just do it before splitting to train and test as follows: all_data['text'] = all_data['text'].apply(clean_text) ? $\endgroup$
    – Ben
    Oct 7, 2022 at 14:04
  • 3
    $\begingroup$ In this case you can but it is important to remember that preprocessing should be done separately for training and test dataset so you don't cause a leak. For example for normalizing you should calculate stats on train dataset only and then use that to normalize test data. In your case you can preprocess all data at the same time but you should know why is possible here (because your preprocessing is independent on other data in the dataset) and when to avoid it. $\endgroup$
    – aEmQy01b
    Oct 7, 2022 at 14:07
  • $\begingroup$ What do you mean by "normalizing"? If I understand you correctly, the only additional place I should apply the clean_text on the test data will be upon actual prediction data? $\endgroup$
    – Ben
    Oct 7, 2022 at 14:11
10
$\begingroup$

There are multiple possible reasons.

  • First, accuracy is certainly not a good evaluation measure for a multiclass problem, unless the dataset is balanced but I doubt it. You should use precision/recall/f1-score, and probably also observe the confusion matrix. It's possible that your results are actually not even meaningful.
  • Preprocessing text data is not "one size fits all", some forms of preprocessing are adapted for some particular task and/or some data. It's also perfectly normal that some tasks don't require any preprocessing. One should design and experiment specifically for a task/dataset, not apply some a specific method blindly.
  • Similarly, preprocessing should be compatible with the feature design: how many feaures are used, how many instances, which algorithm? These can have a huge impact on performance.
$\endgroup$
6
  • $\begingroup$ I actually have only 1 feature called 'text' $\endgroup$
    – Ben
    Oct 6, 2022 at 13:38
  • $\begingroup$ @Ben you're confusing the input data format with the actual features fed to the algorithm. Text must be represented as a set of features, usually bag of words in traditional ML. I assume you're using CountVectorizer or TfidfVectorizer, right? This transforms your text into features, and generally you should not use the least frequent ones because then you have a high risk of overfitting. $\endgroup$
    – Erwan
    Oct 6, 2022 at 14:48
  • $\begingroup$ I've updated my question with additional info $\endgroup$
    – Ben
    Oct 6, 2022 at 19:57
  • $\begingroup$ It's a bit confusing, I don't see the part where you apply the preprocessing: as mentioned by aEmQy01b, in case it's not applied in the same way to the training and test set it would explain a lot. But also there are other issues to check about the NN: is it overfitting in one case or the other? Is there some huge majority class causing a bias? observing confusion matrix in both cases could help. And I'm probably forgetting a few other things to investigate. $\endgroup$
    – Erwan
    Oct 7, 2022 at 12:49
  • $\begingroup$ Added my comment to him as well... I'm unsure where (if any) such stage should occur. $\endgroup$
    – Ben
    Oct 7, 2022 at 13:50
2
$\begingroup$

Alternatively, it might just be that you were heavily overfitting your training model. Removing features, ie cleaning text, makes it harder for the model to overfit. So this is actually helping your model. To know this, you'd need to look at validation accuracy.

$\endgroup$
2
  • $\begingroup$ I'm looking at val_accuracy and it got worse after stemming but better without it $\endgroup$
    – Ben
    Oct 6, 2022 at 20:32
  • $\begingroup$ Can you share the train and val metrics with and without stemming? In addition, what accuracy metric you used? $\endgroup$
    – user70889
    Oct 7, 2022 at 13:02
2
$\begingroup$

In my experience, I've seen it many times that text preprocessing (specially stemming) could worsen your predictions (sometimes it works, and sometimes it doesn't. It depends on the data you're working with). The affirmation "Text preprocessing is a must and will improve your model" is just for some textbooks/medium posts.

$\endgroup$
2
  • 1
    $\begingroup$ I don't think any textbook pretends that text preprocessing is a must. $\endgroup$
    – Erwan
    Oct 6, 2022 at 12:55
  • $\begingroup$ Yeah not every textbook/medium note says that, but is something often said. I don't know how to express my self but it is like "if you want to improve your model try neural networks" , this is just some fancy statement to give you some hope. $\endgroup$
    – Tom
    Oct 6, 2022 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.