I am working on a multiclass classification problem. I want to know whether my model is overfitting or underfitting. I am learning how to plot learning curves and have 4 doubts.
1.) Is the ordering of steps correct as I have done below, i.e scaling
, CV
(baseline model), learning curve
, hyperparameter tuning
, fitting model and predicting
and then learning curve
again to determine if my model is overfitting or underfitting?
2.) Based on the plot, is my model overfitting, as from my confusion matrix
and classification report
I am getting near perfect scores. (sorry I had to upload an image as there was no other choice!)
3.) The plot changes every time I run the model. How, then can I get a appropriate measure of my model?
This is the work I have done so far:
train_x, test_x, train_y, test_y = train_test_split(data2, y, test_size = 0.2, random_state = 69, stratify = y)
cross_val_model = cross_val_score(pipe, train_x, train_y, cv = 5,
n_jobs = -1, scoring = 'f1_macro')
s_score = cross_val_model.mean()
train_sizes, train_score, test_score = learning_curve(pipe, train_x,
train_y, cv = 5,
train_sizes =
np.linspace(0.1, 1.0, 10),
n_jobs = -1, scoring =
'f1_macro')
train_mean = np.mean(train_score, axis = 1)
test_mean = np.mean(test_score, axis = 1)
plt.plot(train_sizes, train_mean, color = 'blue', marker =
'o',markersize = 5, label = 'training acc')
plt.plot(train_sizes, test_mean, color = 'red', marker='+', markersize =
5, label = 'validation acc')
plt.title('learning curve')
plt.xlabel('training data size')
plt.ylabel('model f1 score')
plt.legend()
plt.grid()
plt.show()
def objective(trial):
model__max_depth = trial.suggest_int('model__max_depth', 2, 32)
model__max_leaf_nodes = trial.suggest_int('model__max_leaf_nodes',
50, 500)
model__max_features = trial.suggest_float('model__max_features', 0.0,
1.0)
model__min_samples_leaf =
trial.suggest_int('model__min_samples_leaf', 1, 50)
params = {'model__max_depth' : model__max_depth,
'model__max_leaf_nodes' : model__max_leaf_nodes,
'model__max_features' : model__max_features,
'model__min_samples_leaf' : model__min_samples_leaf}
pipe.set_params(**params)
return np.mean(cross_val_score(pipe, train_x, train_y,
cv = 5, n_jobs = -1, scoring =
'f1_macro'))
dtr_study = optuna.create_study(direction = 'maximize')
dtr_study.optimize(objective, n_trials = 10)
pipe.set_params(**dtr_study.best_params)
pipe.fit(train_x, train_y)
pred = pipe.predict(test_x)
c_matrix = confusion_matrix(test_y, pred)
c_report = classification_report(test_y, pred)
train_sizes, train_score, test_score = learning_curve(pipe, train_x,
train_y, cv = 5,
train_sizes =
np.linspace(0.1, 1.0, 10),
n_jobs = -1, scoring =
'f1_macro')
train_mean = np.mean(train_score, axis = 1)
test_mean = np.mean(test_score, axis = 1)
plt.plot(train_sizes, train_mean, color = 'blue', marker =
'o',markersize = 5, label = 'training acc')
plt.plot(train_sizes, test_mean, color = 'red', marker='+', markersize =
5, label = 'validation acc')
plt.title('learning curve')
plt.xlabel('training data size')
plt.ylabel('model f1 score')
plt.grid()
plt.show()