Learning curves

Question

I am working on a multiclass classification problem. I want to know whether my model is overfitting or underfitting. I am learning how to plot learning curves and have 4 doubts.

1.) Is the ordering of steps correct as I have done below, i.e scaling, CV (baseline model), learning curve, hyperparameter tuning, fitting model and predicting and then learning curve again to determine if my model is overfitting or underfitting?

2.) Based on the plot, is my model overfitting, as from my confusion matrix and classification report I am getting near perfect scores. (sorry I had to upload an image as there was no other choice!)

3.) The plot changes every time I run the model. How, then can I get a appropriate measure of my model?

This is the work I have done so far:

train_x, test_x, train_y, test_y = train_test_split(data2, y, test_size = 0.2, random_state = 69, stratify = y)


cross_val_model = cross_val_score(pipe, train_x, train_y, cv = 5,
                              n_jobs = -1, scoring = 'f1_macro')
s_score = cross_val_model.mean()

train_sizes, train_score, test_score = learning_curve(pipe, train_x, 
train_y, cv = 5,
                                                 train_sizes = 
np.linspace(0.1, 1.0, 10),
                                                 n_jobs = -1, scoring = 
'f1_macro')
train_mean = np.mean(train_score, axis = 1)
test_mean = np.mean(test_score, axis = 1)

plt.plot(train_sizes, train_mean, color = 'blue', marker = 
'o',markersize = 5, label = 'training acc')
plt.plot(train_sizes, test_mean, color = 'red', marker='+', markersize = 
5, label = 'validation acc')
plt.title('learning curve')
plt.xlabel('training data size')
plt.ylabel('model f1 score')
plt.legend()
plt.grid()
plt.show()

def objective(trial):

   model__max_depth = trial.suggest_int('model__max_depth', 2, 32)
   model__max_leaf_nodes = trial.suggest_int('model__max_leaf_nodes', 
50, 500)
   model__max_features = trial.suggest_float('model__max_features', 0.0, 
1.0)
   model__min_samples_leaf = 
trial.suggest_int('model__min_samples_leaf', 1, 50)

   params = {'model__max_depth' : model__max_depth, 
          'model__max_leaf_nodes' : model__max_leaf_nodes,
          'model__max_features' : model__max_features,
          'model__min_samples_leaf' : model__min_samples_leaf}

   pipe.set_params(**params)

   return np.mean(cross_val_score(pipe, train_x, train_y,
                                 cv = 5, n_jobs = -1, scoring = 
  'f1_macro'))


dtr_study = optuna.create_study(direction = 'maximize')
dtr_study.optimize(objective, n_trials = 10)

pipe.set_params(**dtr_study.best_params)
pipe.fit(train_x, train_y)
pred = pipe.predict(test_x)
c_matrix = confusion_matrix(test_y, pred)
c_report = classification_report(test_y, pred)

train_sizes, train_score, test_score = learning_curve(pipe, train_x, 
train_y, cv = 5,
                                                 train_sizes = 
np.linspace(0.1, 1.0, 10),
                                                 n_jobs = -1, scoring = 
'f1_macro')
train_mean = np.mean(train_score, axis = 1)
test_mean = np.mean(test_score, axis = 1)

plt.plot(train_sizes, train_mean, color = 'blue', marker = 
'o',markersize = 5, label = 'training acc')
plt.plot(train_sizes, test_mean, color = 'red', marker='+', markersize = 
5, label = 'validation acc')
plt.title('learning curve')
plt.xlabel('training data size')
plt.ylabel('model f1 score')
plt.grid()
plt.show()

Answer 1

Normally the learning curves use

• X axis = Number of iterations of the model
• Y axis = How good the model is, where good depends on your loss function (in your case, that would be the f1-score)

In your case you seem to be using the size of your training data.

Think about it: The learning curve shows how much better your model gets over time, not over data. Normally more data means better models, but that is not what you want to measure with your learning curve.