2
$\begingroup$

I want to do a multi class classification with 6 classes. Whole dataset has 12750 and 56 features samples, so every class has 2125 samples. Before prediction I reduces amount of outliers by winsorization (for 1 and 99 percentile) and I reduced skewness in features which has more than 1 and less than -1 skewness by Yeo-Johnson transformation and I got dataset:

https://i.stack.imgur.com/miy8i.png

Later, of course, I splitted dataset for 80% of training data and 20% of test data and I standardised training data. I tried to use random forest, xgboost and decision tree classifiers, but I have almost 100% accuracy on training set and 20-21% accuracy on test set. Methods like increasing n_estimators doesn't help.

So, my questions are:

How can I reduce this overfitting? Is it a problem with dataset (Should I reduce number of features something like that?) or with classificators (Are they too weak for this problem?)

Is the dataset too small for this problem (Should I add more samples by method like SMOTE?)? Do classes have too less samples to good work?

Is it possible to get at least 60% accuracy after tuning hyperparameters (e.g. by method like GridSearchCV)?

P.S. I will add that correlations with target value are very poor (max +- 6%) and I see that feature importances from random forest have values from 0.0 to 0.03. I don't know if this is a normal situation.

P.S.2 I tried to change n_estimators parameters (values from 5 to 1500) and max_depth (from 1 to 100) and I can see very poor change in test accuracy (+-3%)

$\endgroup$

2 Answers 2

2
$\begingroup$

There are few things I can think of:

  1. Did you use stratified split for train/test sets?
  2. Try using validation set and doing early stopping based on validation loss.
  3. Try using N-fold cross validation.
  4. Add regularization to your models and use GridSearchCV to play with their parameters.

It is difficult to say anything more in particular without looking at your data.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer. I tried statified split (in train_test_split() from sklearn parameter stratified=y), but It didn't make difference. Also I tried to make many tests in GridSearchCV, especially changing max_features (values from 1 to 50) and max_depth (values from 1 to 30). I read that regularization in random forest is reducing max_depth, but it made a little increase (~2-3%). I assumed that k-fold cross validation is in GridSearchCV, so I didn't make it separately $\endgroup$
    – jared
    Apr 26 at 11:39
  • $\begingroup$ Yes, k-fold validation is indeed in GridSearchCV. Have you tried logistic regression on the same data, is there a similar gap between between train/test accuracy? For a single decision tree, try this example from sklearn. in XGBoost use "early_stopping_rounds" during training, some of the regularization parameters I'd try are colsample_by*, lambda, alpha $\endgroup$
    – Aramakus
    Apr 26 at 12:29
2
$\begingroup$

A few comments:

  • First, you say that every class has 2125 samples: I'm surprised that it's perfectly balanced, is this the real distribution of the data? It would be a mistake to artificially balance the classes, especially before the split between training and test set.
  • In principle the preprocessing should be done based only on the training set, but this is unlikely to cause a serious problem here.
  • Overfitting happens when the model captures things which happen by chance in the training data instead of real patterns, either because it doesn't have enough instances or because it's trying to find too subtle details. Generating artificial instances doesn't always work, because it breaks the distribution of the data. So the simplest solution is to make the model simpler: reduce the number of features, reduce the number of parameters (e.g depth of the tree), etc.
  • Keep in mind that a random baseline classifier would obtain only 16.6% accuracy with 6 classes. So 20% is not that bad, it depends how difficult the task is, i.e. how much the features help to determine the target. There's no guarantee at all that 60% can be reached with any data and any task. However it must be possible not to overfit the model, i.e. to obtain quite similar performance on the training and test set.
$\endgroup$
2
  • $\begingroup$ Thank you for you answer. At the beginning classes had proportion 3088:2125:2361:3027:2203:2160 (together almost 15k samples), but I think it is too unbalanced, so I decided to make undersampling to 2125 samples in every class (let me know if this is the right approach). I know that simplifying model can reduce overfitting and I tried to check many configurations of hyperparameters in GridSearchCV, especially max_features (values from 1 to 50) and max_depth (values from 1 to 30) $\endgroup$
    – jared
    Apr 26 at 11:32
  • 1
    $\begingroup$ @jared your classes have very little imbalance actually, high imbalance is when some class has at least 3 times more instances than another. So I think it would be better not do anything about the distribution, and in any case the model should be evaluated on the original distribution. $\endgroup$
    – Erwan
    May 1 at 14:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.