I'm trying to implement the validation curve based on this SKLearn tutorial. On the site, it shows how based on the parameters the model goes from under- to overfitted, finding the optimal parameter in the middle. My implementation you can see below, but my curve is weird - the train and test scores seem not do differ at all. What does this mean? Am I doing something incorrectly? My inputs X
and y
are shaped (266531, 23) and (266531,).
The curve looks like this:
And my code is:
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.learning_curve import validation_curve
import psutil
np.random.seed(0)
X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()
indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]
param_range = np.arange(1, 41, 2)
train_scores, test_scores = validation_curve(
DecisionTreeClassifier(class_weight='balanced'), X, y, param_name="max_depth", cv=10,
param_range=param_range,n_jobs=psutil.cpu_count(),
scoring="accuracy")
train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)
plt.title("Validation Curve with DecisionTree")
plt.xlabel("max_depth")
plt.ylabel("Score")
plt.ylim(0.0, 1.1)
plt.plot(param_range, train_scores_mean, label="Training score",
color="r")
plt.plot(param_range, test_scores_mean, label="Cross-validation score",
color="g")
plt.legend(loc="best")
plt.xticks(param_range)
plt.show()
UPDATE
A comment suggested I made X
and y
identical. This is not the case. What else might cause the validation curve to look like this? I don't think it's right.
n_jobs=-1
$\endgroup$