Validation curve unlike SKLearn sample

Question

I'm trying to implement the validation curve based on this SKLearn tutorial. On the site, it shows how based on the parameters the model goes from under- to overfitted, finding the optimal parameter in the middle. My implementation you can see below, but my curve is weird - the train and test scores seem not do differ at all. What does this mean? Am I doing something incorrectly? My inputs X and y are shaped (266531, 23) and (266531,).

The curve looks like this:

And my code is:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.learning_curve import validation_curve
import psutil

np.random.seed(0)

X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()
indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

param_range = np.arange(1, 41, 2)
train_scores, test_scores = validation_curve(
    DecisionTreeClassifier(class_weight='balanced'), X, y, param_name="max_depth", cv=10, 
    param_range=param_range,n_jobs=psutil.cpu_count(),
    scoring="accuracy")

train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)
plt.title("Validation Curve with DecisionTree")
plt.xlabel("max_depth")
plt.ylabel("Score")
plt.ylim(0.0, 1.1)
plt.plot(param_range, train_scores_mean, label="Training score",
             color="r")
plt.plot(param_range, test_scores_mean, label="Cross-validation score",
             color="g")

plt.legend(loc="best")
plt.xticks(param_range)
plt.show()

UPDATE

A comment suggested I made X and y identical. This is not the case. What else might cause the validation curve to look like this? I don't think it's right.

Answer 1

This is exactly your code just with digits data:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import load_digits
from sklearn.svm import SVC
from sklearn.learning_curve import validation_curve
import psutil
from sklearn.tree import DecisionTreeClassifier

np.random.seed(0)

# X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()


digits = load_digits()
X, y = digits.data, digits.target

indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

param_range = np.arange(1, 41, 2)
train_scores, test_scores = validation_curve(
    DecisionTreeClassifier(class_weight='balanced'), X, y, param_name="max_depth", cv=10, 
    param_range=param_range,n_jobs=psutil.cpu_count(),
    scoring="accuracy")

train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)
plt.figure(figsize=(10,10))
plt.title("Validation Curve with DecisionTree")
plt.xlabel("max_depth")
plt.ylabel("Score")
plt.ylim(0.0, 1.1)
#plt.fill_between(param_range, train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, alpha=0.2, color="r")
plt.plot(param_range, train_scores_mean, label="Training score",
             color="r")
#plt.fill_between(param_range, test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, alpha=0.2, color="g")
plt.plot(param_range, test_scores_mean, label="Cross-validation score",
             color="g")

plt.legend(loc="best")
plt.xticks(param_range)
plt.savefig('DSSE.jpg')
plt.show()

And this is the result:

So whatever the problem is, it's in your data that you didn't explain how you created it. Either double check it or post it here so your question can be answered. I assume somewhere in data preparation you made X and Y identical by mistake.