# Validation curve unlike SKLearn sample

I'm trying to implement the validation curve based on this SKLearn tutorial. On the site, it shows how based on the parameters the model goes from under- to overfitted, finding the optimal parameter in the middle. My implementation you can see below, but my curve is weird - the train and test scores seem not do differ at all. What does this mean? Am I doing something incorrectly? My inputs X and y are shaped (266531, 23) and (266531,).

The curve looks like this:

And my code is:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.svm import SVC
from sklearn.learning_curve import validation_curve
import psutil

np.random.seed(0)

X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()
indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

param_range = np.arange(1, 41, 2)
train_scores, test_scores = validation_curve(
DecisionTreeClassifier(class_weight='balanced'), X, y, param_name="max_depth", cv=10,
param_range=param_range,n_jobs=psutil.cpu_count(),
scoring="accuracy")

train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)
plt.title("Validation Curve with DecisionTree")
plt.xlabel("max_depth")
plt.ylabel("Score")
plt.ylim(0.0, 1.1)
plt.plot(param_range, train_scores_mean, label="Training score",
color="r")
plt.plot(param_range, test_scores_mean, label="Cross-validation score",
color="g")

plt.legend(loc="best")
plt.xticks(param_range)
plt.show()


UPDATE

A comment suggested I made X and y identical. This is not the case. What else might cause the validation curve to look like this? I don't think it's right.

• quick note: if you want to use all available CPUs, just set n_jobs=-1 Jan 23 '18 at 9:06

This is exactly your code just with digits data:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.svm import SVC
from sklearn.learning_curve import validation_curve
import psutil
from sklearn.tree import DecisionTreeClassifier

np.random.seed(0)

# X, y = prepareDataframeX.values, prepareDataframeY.values.ravel()

X, y = digits.data, digits.target

indices = np.arange(y.shape[0])
np.random.shuffle(indices)
X, y = X[indices], y[indices]

param_range = np.arange(1, 41, 2)
train_scores, test_scores = validation_curve(
DecisionTreeClassifier(class_weight='balanced'), X, y, param_name="max_depth", cv=10,
param_range=param_range,n_jobs=psutil.cpu_count(),
scoring="accuracy")

train_scores_mean = np.mean(train_scores, axis=1)
train_scores_std = np.std(train_scores, axis=1)
test_scores_mean = np.mean(test_scores, axis=1)
test_scores_std = np.std(test_scores, axis=1)
plt.figure(figsize=(10,10))
plt.title("Validation Curve with DecisionTree")
plt.xlabel("max_depth")
plt.ylabel("Score")
plt.ylim(0.0, 1.1)
#plt.fill_between(param_range, train_scores_mean - train_scores_std, train_scores_mean + train_scores_std, alpha=0.2, color="r")
plt.plot(param_range, train_scores_mean, label="Training score",
color="r")
#plt.fill_between(param_range, test_scores_mean - test_scores_std, test_scores_mean + test_scores_std, alpha=0.2, color="g")
plt.plot(param_range, test_scores_mean, label="Cross-validation score",
color="g")

plt.legend(loc="best")
plt.xticks(param_range)
plt.savefig('DSSE.jpg')
plt.show()


And this is the result:

So whatever the problem is, it's in your data that you didn't explain how you created it. Either double check it or post it here so your question can be answered. I assume somewhere in data preparation you made X and Y identical by mistake.

• I'll update the post: X and y are NOT identical. Any other things that might cause this? Feb 6 '18 at 11:41
• @Kasra Manshaei . Could you please let me know how to plot validation curve for class weight? Apr 14 '18 at 3:01
• @Ebrahimi What do you mean by Class Weight? Apr 14 '18 at 9:02
• @KasraManshaei Thanks. In your code: class_weight='balanced' but I want to plot validation curve if class_weight will be: param_range2=[{ 0:1, 1:6 },{ 0:1, 1:4 },{ 0:1, 1:5.5 },{ 0:1, 1:4.5 },{ 0:1, 1:5 }]. However, TypeError: float() argument must be a string or a number, not 'dict'is produced. Apr 14 '18 at 11:27