Why my sentiment analysis model is overfitting?

Question

The task is to predict sentiment from 1 to 10 based on Russian reviews. The training data size is 20000 records, of which 1000 were preserved as a validation set. The preprocessing steps included punctuation removal, digit removal, Latin character removal, stopword removal, and lemmatization. Since the data was imbalanced, I decided to downsample it. After that, TF-IDF vectorization was applied. At the end, I got this training dataset:

The next step was the validation set TF-IDF transformation:

As a classifier model, I chose MultinomialNB (I read it is useful for text classification tasks and sparse data). The training data fit was pretty quick:

# TODO: create a Multinomial Naive Bayes Classificator

clf = MultinomialNB(force_alpha=True)
clf.fit(X_res, y_res.values.ravel())

But the problem was in model evaluation part:

# TODO: model evaluation

print(clf.score(X_res, y_res.values.ravel()))
print(clf.score(X_val, y_val.values.ravel()))
y_pred = clf.predict(X_val)
print(precision_recall_fscore_support(y_val, y_pred, average='macro'))

Output:

0.9352409638554217
0.222
(0.17081898127154763, 0.1893033502842826, 0.16303596541199034, None)

It is obvious that the model is overfitting, but what do I do? I tried to use SVC, KNeighborsClassifier, DecisionTreeClassifier, RandomForestClassifier, and GaussianNB, but everything remained the same. I tried to play around with the MultinomialNB hyperparameter alpha but force_alpha=True option is the best so far.

Answer 1

There might be multiple reasons that might be the reason for overfitting some of which are:

1.) Scaling the data

2.) You have not mentioned which parameter values you have selected in the Tfidf vectorizer. Some of them might help to reduce overfitting. ngram_range and max_features are 2 which you can play around with.

3.) Make sure you are using fit_transform on the train set only and not on the test set for both tfidf and scaling. Use only transform for the test set.

4.) Try to tune the hyperparameters of other models such as RandomForest and SVC.

5.) Use other word embedding techniques such as Word2Vec, Glove or Fasttext as they capture the word context as well as opposed to just the word frequency (which is happening in the case of tfidf).

6.) Try different models. You are just testing 4-5 models when in fact there are so many classification models out there. Try as many as you can to see which one gives the best result.

7.) Last but not the least,increase the data size. Since you are down sampling the data (I don't know by how much), this also might be a factor in overfitting.

Try to implement all of the above points and let me know whether results improve.

Cheers!