0
$\begingroup$

The task is to predict sentiment from 1 to 10 based on Russian reviews. The training data size is 20000 records, of which 1000 were preserved as a validation set. The preprocessing steps included punctuation removal, digit removal, Latin character removal, stopword removal, and lemmatization. Since the data was imbalanced, I decided to downsample it. After that, TF-IDF vectorization was applied. At the end, I got this training dataset:

enter image description here

The next step was the validation set TF-IDF transformation:

enter image description here

As a classifier model, I chose MultinomialNB (I read it is useful for text classification tasks and sparse data). The training data fit was pretty quick:

# TODO: create a Multinomial Naive Bayes Classificator

clf = MultinomialNB(force_alpha=True)
clf.fit(X_res, y_res.values.ravel())

But the problem was in model evaluation part:

# TODO: model evaluation

print(clf.score(X_res, y_res.values.ravel()))
print(clf.score(X_val, y_val.values.ravel()))
y_pred = clf.predict(X_val)
print(precision_recall_fscore_support(y_val, y_pred, average='macro'))

Output:

0.9352409638554217
0.222
(0.17081898127154763, 0.1893033502842826, 0.16303596541199034, None)

It is obvious that the model is overfitting, but what do I do? I tried to use SVC, KNeighborsClassifier, DecisionTreeClassifier, RandomForestClassifier, and GaussianNB, but everything remained the same. I tried to play around with the MultinomialNB hyperparameter alpha but force_alpha=True option is the best so far.

$\endgroup$

1 Answer 1

1
$\begingroup$

There might be multiple reasons that might be the reason for overfitting some of which are:

1.) Scaling the data

2.) You have not mentioned which parameter values you have selected in the Tfidf vectorizer. Some of them might help to reduce overfitting. ngram_range and max_features are 2 which you can play around with.

3.) Make sure you are using fit_transform on the train set only and not on the test set for both tfidf and scaling. Use only transform for the test set.

4.) Try to tune the hyperparameters of other models such as RandomForest and SVC.

5.) Use other word embedding techniques such as Word2Vec, Glove or Fasttext as they capture the word context as well as opposed to just the word frequency (which is happening in the case of tfidf).

6.) Try different models. You are just testing 4-5 models when in fact there are so many classification models out there. Try as many as you can to see which one gives the best result.

7.) Last but not the least,increase the data size. Since you are down sampling the data (I don't know by how much), this also might be a factor in overfitting.

Try to implement all of the above points and let me know whether results improve.

Cheers!

$\endgroup$
10
  • $\begingroup$ 2) For TF-IDF I used default parameters. But I noticed that when I set min_df, max_df or max_features the training set accuracy get worse than applying default parameters $\endgroup$ May 25, 2023 at 10:58
  • $\begingroup$ 1) Do you think the scaling might be one of the solutions? Because TF-IDF values already scaled between 0-1 $\endgroup$ May 25, 2023 at 11:00
  • $\begingroup$ 3) Yes, I used fit_transform on train set and transform on test. I did such mistake before, but even though I noticed it and used it correctly, the model performance remained the same giving me a low test accuracy $\endgroup$ May 25, 2023 at 11:03
  • $\begingroup$ @RenatAbdrakhmanov I know Tfidf gives output between 0 and 1. Scaling might or might not improve the results. Try and see. I think the language might be the problem. That is why I said you should use other embedding techniques as they are trianed on multiple languages (including Russian). They might better capture the context of the text as tfidf is just calculating the word frequency and not the context. $\endgroup$
    – spectre
    May 25, 2023 at 11:04
  • $\begingroup$ 5) I am about to use Word2Vec. I found that spaCy does word emebeddings, so if I will apply it to every sentence in dataset, then I can use vectors as my features? $\endgroup$ May 26, 2023 at 4:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.