svm.LinearSVC: larger max_iter number doesn't always increase the accuracy/precision/recall

Question

Background:

• Supervised machine learning
• Data shape
  • 10+ features, target = 1 or 0 only, 100,000+ samples (so should be no issue of over-sampling)

• 80% training, 20% testing

  train_test_split(X_train, Y_train, test_size=0.2)

• Use svm.LinearSVC(max_iter = N).fit( ) to train labelled data

  • Scaling not applied yet (all feature values are around 0-100 (float64))
  • Other parameters (e.g., c = ) use default value

Results:

print("Accuracy:", metrics.accuracy_score(y_test, y_pred))
print("Precision:", metrics.precision_score(y_test, y_pred))
print("Recall:", metrics.recall_score(y_test, y_pred))

Question:

I increased max_iter = from 1,000 to 10,000 and 100,000, but above 3 scores don't show a trend of increments. The score of 10,000 is worse than 1,000 and 100,000.

For example, max_iter = 100,000

Accuracy: 0.9728548424200598
Precision: 0.9669730040206778
Recall: 0.9653096330275229

max_iter = 10,000

Accuracy: 0.9197914270378038
Precision: 0.9886761615689937
Recall: 0.8093463302752294

max_iter = 1,000

Accuracy: 0.9838969404186796
Precision: 0.964741810105497
Recall: 0.9962729357798165

1. What could be the reason?
2. Do I need to test different max_iter values and select the best performance? For example, use GridSearchCV( )

Answer 1

When trying to find the optimum number of iterations it's normally quite useful to visualise how the increasing iteration effect the accuracy (can identify over-fitting and when you should stop fitting).

# Import libraries used
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.metrics import accuracy_score
from sklearn.svm import SVC

# Create a template lit to store accuracies
acc = []

# Iterate along a logarithmically spaced ranged
for i in np.logspace(0,5, num = 6):
    # Print out the number of iterations to use for the current loop
    print('Training model with iterations: ', i)
    # Create an SVC algorithm with the number of iterations for the current loop
    svc = SVC(solver = 'lbfgs', multi_class = 'auto', max_iter = i, class_weight='balanced')
    # Fit the algorithm to the data
    svc.fit(X_train, Y_train)
    # Append the current accuracy score to the template list
    acc.append(accuracy_score(Y_test, logreg.predict(X_test)) * 100)

# Convert the accuracy list to a series
acc = pd.Series(acc, index = np.logspace(0,5, num = 6))
# Set the plot size
plt.figure(figsize = (15,10))
# Set the plot title
title = 'Graph to show the accuracy of the SVC model as number of iterations increases\nfinal accuracy: ' + str(acc.iloc[-1])
plt.title(title)
# Set the xlabel and ylabel
plt.xlabel('Number of iterations')
plt.ylabel('Accuracy score')
# Plot the graph
acc.plot.line()
plt.show()

This will produce a graph where the number of iterations has been logarithmic increased (note that it might take a bit of time experimenting with np.logspace to create whole number iteration steps).

If the accuracy increases then keep following the trend, if it plateus stop theres probably no point wasting your time, if it drops go back to the highest value (you've over-fitted to the training data).