0
$\begingroup$

I am working on a multiclass classification problem. I want to know whether my model is overfitting or underfitting. I am learning how to plot learning curves and have 4 doubts.

1.) Is the ordering of steps correct as I have done below, i.e scaling, CV (baseline model), learning curve, hyperparameter tuning, fitting model and predicting and then learning curve again to determine if my model is overfitting or underfitting?

2.) Based on the plot, is my model overfitting, as from my confusion matrix and classification report I am getting near perfect scores. (sorry I had to upload an image as there was no other choice!)

3.) The plot changes every time I run the model. How, then can I get a appropriate measure of my model?

This is the work I have done so far:

train_x, test_x, train_y, test_y = train_test_split(data2, y, test_size = 0.2, random_state = 69, stratify = y)


cross_val_model = cross_val_score(pipe, train_x, train_y, cv = 5,
                              n_jobs = -1, scoring = 'f1_macro')
s_score = cross_val_model.mean()

train_sizes, train_score, test_score = learning_curve(pipe, train_x, 
train_y, cv = 5,
                                                 train_sizes = 
np.linspace(0.1, 1.0, 10),
                                                 n_jobs = -1, scoring = 
'f1_macro')
train_mean = np.mean(train_score, axis = 1)
test_mean = np.mean(test_score, axis = 1)

plt.plot(train_sizes, train_mean, color = 'blue', marker = 
'o',markersize = 5, label = 'training acc')
plt.plot(train_sizes, test_mean, color = 'red', marker='+', markersize = 
5, label = 'validation acc')
plt.title('learning curve')
plt.xlabel('training data size')
plt.ylabel('model f1 score')
plt.legend()
plt.grid()
plt.show()

enter image description here

def objective(trial):

   model__max_depth = trial.suggest_int('model__max_depth', 2, 32)
   model__max_leaf_nodes = trial.suggest_int('model__max_leaf_nodes', 
50, 500)
   model__max_features = trial.suggest_float('model__max_features', 0.0, 
1.0)
   model__min_samples_leaf = 
trial.suggest_int('model__min_samples_leaf', 1, 50)

   params = {'model__max_depth' : model__max_depth, 
          'model__max_leaf_nodes' : model__max_leaf_nodes,
          'model__max_features' : model__max_features,
          'model__min_samples_leaf' : model__min_samples_leaf}

   pipe.set_params(**params)

   return np.mean(cross_val_score(pipe, train_x, train_y,
                                 cv = 5, n_jobs = -1, scoring = 
  'f1_macro'))


dtr_study = optuna.create_study(direction = 'maximize')
dtr_study.optimize(objective, n_trials = 10)

pipe.set_params(**dtr_study.best_params)
pipe.fit(train_x, train_y)
pred = pipe.predict(test_x)
c_matrix = confusion_matrix(test_y, pred)
c_report = classification_report(test_y, pred)

train_sizes, train_score, test_score = learning_curve(pipe, train_x, 
train_y, cv = 5,
                                                 train_sizes = 
np.linspace(0.1, 1.0, 10),
                                                 n_jobs = -1, scoring = 
'f1_macro')
train_mean = np.mean(train_score, axis = 1)
test_mean = np.mean(test_score, axis = 1)

plt.plot(train_sizes, train_mean, color = 'blue', marker = 
'o',markersize = 5, label = 'training acc')
plt.plot(train_sizes, test_mean, color = 'red', marker='+', markersize = 
5, label = 'validation acc')
plt.title('learning curve')
plt.xlabel('training data size')
plt.ylabel('model f1 score')
plt.grid()
plt.show()

enter image description here

$\endgroup$

1 Answer 1

0
$\begingroup$

Normally the learning curves use

  • X axis = Number of iterations of the model
  • Y axis = How good the model is, where good depends on your loss function (in your case, that would be the f1-score)

In your case you seem to be using the size of your training data.

Think about it: The learning curve shows how much better your model gets over time, not over data. Normally more data means better models, but that is not what you want to measure with your learning curve.

$\endgroup$
3
  • $\begingroup$ "The learning curve shows how much better your model gets over time, not over data." I disagree with this statement. In contrast to the general term of a learning curve, a learning curve in ML does exactly that: it plots the amount of data vs. a score. $\endgroup$
    – Jonathan
    Sep 9, 2021 at 7:13
  • $\begingroup$ @Juan Antonio Gomez Moriano Yes theoretically I can plot any quantity on the x axis but that is not the question I have. Can you answer from the 3 doubts I have listed in the question? $\endgroup$
    – spectre
    Sep 9, 2021 at 11:13
  • $\begingroup$ @Sammy Ok. So I have plotted 2 curves, one prior to tuning and the other after tuning. Is it the correct methodology? $\endgroup$
    – spectre
    Sep 9, 2021 at 11:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.