0
$\begingroup$

I have been reading about the validation_curve function from scikit learn. When I run this it takes too long. Therefore, I am plotting the results from grid search instead, which seems to be far quicker. However, when I plot using the code from: https://matthewbilyeu.com/blog/2019-02-05/validation-curve-plot-from-gridsearchcv-results.

I get:

enter image description here

Why are the lines so close together? Shouldn't I get a curve instead? More generally, how can I tell if my model is under-fitting or overfitting? How can I plot the overfit/under-fit areas on my plot.

edit: i am trying to predict if a client will default on a loan or not so 1 for yes 0 for no. the data are financials. also the data i have used is about 100k samples

Improve this question
$\endgroup$
3
  • $\begingroup$ At first glance it looks like you are using a small dataset that is easy to model. Regardless of the hyperparameters you choose, xgboost easily finds an optimal solution. Cross validated performance is a little lower as you have a few tricky examples that don’t fit the pattern described by the rest of the dataset, and when these are in the hold-out set in cross validation the model performs worse. Could you please describe your dataset and tell us what you are trying to predict? $\endgroup$
    – Nicholas James Bailey
    Jun 18, 2020 at 20:49
  • $\begingroup$ thanks i have updated it $\endgroup$
    – Maths12
    Jun 19, 2020 at 7:36
  • $\begingroup$ I think we'd need the grid search and parameter space definitions, as well as how you called the blogs function. $\endgroup$
    – Ben Reiniger ♦
    Aug 1, 2022 at 17:02

2 Answers 2

Reset to default
0
$\begingroup$

You can understand whether your model is overfitting or underfitting by the difference in the graph of your train and validation score. If your train score(performance not loss) is low and so is val score then your model is underfitting. On the other hand if your model is overfiiting you will have high training accuracy but your validation score will be low and the train and val graph will be far from each other. A perfect model just has high training score with the validation curve as close as possible and the two graphs will be very close like the graph you provided.

Improve this answer
$\endgroup$
0
$\begingroup$

It looks like there are very few loan defaults in your dataset, so xgboost is learning to predict 0 for all inputs regardless of the hyperparameters you choose. Try sampling from your data so that the classes are more balanced (e.g. >10% of your data points are defaults) and see what happens.

Other options for dealing with highly imbalanced classes can be found here: https://www.analyticsvidhya.com/blog/2017/03/imbalanced-data-classification/

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.