I have a dataset and I have built an XGBClassifier model from it. Without hyperparameter tuning, the model performs fairly well in training but on test which have some signs of overfitting (train accuracy: 90, test accuracy: 80). After hyperparameter tuning (set the L1, L2, eta, gamma, learning_rate), the model seems to be better in terms of overfitting but the accuracy has been reduced (train accuracy: 70, test accuracy: 69). I have utilized scale_pos_weight to balance the binary target variable. Any suggestion on how could I further enhance the accuracy while not overfitting/ underfitting the model? I have tried all possible ways that I know as a beginner.
6
-
$\begingroup$ I would be less concerned about the difference between train and test accuracy than about the train accuracy. What are your concerns about the difference between train and test accuracy? // Accuracy is more problematic than most people realize, even with perfect class balance. $\endgroup$– DaveJun 22, 2022 at 16:13
-
$\begingroup$ Like Dave pointed out, is accuracy the right metric to use for your business problem? Is the cut off value for to determine the accuracy optimized for your business problem for both models? Looking to improve statistical performance in models you need to look at the features - get more/better features, do additional feature engineering, get more data if applicable. $\endgroup$– CraigJun 22, 2022 at 16:36
-
$\begingroup$ I'm focusing on other aspects such as TN but using accuracy as a reference point for overfitting/ underfitting issues. Using XGB models, should I normalize the distribution of features or drop variables that has colinearity > .8? I read that XGB is quite robust to these issues internally and there is no need to do so. $\endgroup$– Kusisi KaremJun 23, 2022 at 5:36
-
$\begingroup$ Mess with the decision threshold between the two classes. In my experience you can get a decent jump just based on not using .5. $\endgroup$– Michael HigginsJun 23, 2022 at 18:40
-
$\begingroup$ Thanks a lot, Michael $\endgroup$– Kusisi KaremJun 24, 2022 at 4:53
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
The reduction in evaluation metric values between training and test is less important than the overall ability of a model to generalize. In your case, the first model performs much better in absolute terms (80% test accuracy) and that is currently the best model to use.
Enhancing accuracy is a question of the goals of the project. Does the model need to be better to accomplish the goals of the project?
You might have to go beyond hyperparameter fitting to improve model performance. You can try adding more data and better features.
answered Jul 4, 2022 at 18:06