How to check for overfitting with SVM and Iris Data?

Question

I am using machine learning predictions for the sample iris dataset. For instance, I am using the support vector machines (SVMs) from scikit-learn in order to predict the accuracy. However, it returns an accuracy of 1.0. Here is the code I am using:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=51)
svm_model = svm.SVC(kernel='linear', C=1, gamma='auto')
svm_model.fit(X_train,y_train)
predictions = svm_model.predict(X_test)
accuracy_score(predictions, y_test)

How to find out or to measure if this over-fitting or if the model is so good? I assume that its not over-fitting but what are the best ways to validate this?

Answer 1

You check for hints of overfitting by using a training set and a test set (or a training, validation and test set). As others have mentioned, you can either split the data into training and test sets, or use cross-fold validation to get a more accurate assessment of your classifier's performance.

Since your dataset is small, splitting your data into training and test sets isn't recommended. Use cross validation.

This can be done using either the cross_validate or cross_val_score function; the latter providing multiple metrics for evaluation. In addition to test scores the latter also provides fit times and score times.

Using your example;

from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
from sklearn import svm
from sklearn.metrics import accuracy_score

iris = load_iris()
X = iris.data[:, :5]  # we only take the first two features.
y = iris.target

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=51)
svm_model = svm.SVC(kernel='linear', C=1, gamma='auto')
svm_model.fit(X_train,y_train)
predictions = svm_model.predict(X_test)
accuracy_score(predictions, y_test)

raw accuracy: 0.96666666666666667

Using the cross_val_score function, and printing the mean score and 95% confidence interval of the score estimate:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(svm_model, iris.data, iris.target, cv=5)
print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2))

Accuracy: 0.98 (+/- 0.03)

Of course the iris dataset is a toy example. On larger real-world datasets you are likely to see your test error be higher than your training error, with cross-validation providing a lower accuracy than the raw number.

So I wouldn't use the iris dataset to showcase overfitting. Choose a larger, messier dataset, and then you can start working towards reducing the bias and variance of the model (the "causes" of overfitting).

Then you can start exploring tell-tale signs of whether it's a bias problem or a variance problem. See here:

https://www.quora.com/How-many-training-samples-are-needed-to-get-a-reliable-model-in-ML/answer/Sean-McClure-3?srid=zGgv

Answer 2

It might be a problem of over-fitting, or that by just doing a single train / test split isn't giving a reliable estimate of the generalizable error of your SVM.

I'd recommend using KFold validation to check.

from sklearn.model_selection import KFold
import numpy as np
acc_score = []

kf = KFold(n_splits=5)

for train_index, test_index in kf.split(X):

    X_train, X_test = X[train_index], X[test_index]
    y_train, y_test = y[train_index], y[test_index]

    svm_model.fit(X_train,y_train)
    predictions = svm_model.predict(X_test)
    acc_score.append(accuracy_score(predictions, y_test))

np.mean(acc_score)

If the average is still 1.0 then you've done good, but my gut feeling is that your high score is dependent on the cut of the data you're looking at.

Answer 3

Based on here, use sklearn.model_selection.train_test_split(*arrays, **options) in order to split your data into train and test. Train your model on train-split and use the predict method to see the performance on the test data. As an example take a look at the following code which splits the data to two separate groups.

import numpy as np
from sklearn.model_selection import train_test_split
X, y = np.arange(10).reshape((5, 2)), range(5)
X

array([[0, 1],
       [2, 3],
       [4, 5],
       [6, 7],
       [8, 9]])

list(y)
[0, 1, 2, 3, 4]


X_train, X_test, y_train, y_test = train_test_split(
...     X, y, test_size=0.33, random_state=42)
...
X_train
array([[4, 5],
   [0, 1],
   [6, 7]])
y_train
[2, 0, 3]
X_test
array([[2, 3],
   [8, 9]])
y_test
[1, 4]

train_test_split(y, shuffle=False)
[[0, 1, 2], [3, 4]]