sklearn: SGDClassifier yields lower accuracy than LogisticRegression

Question

I'm participating in the kaggle Iceberg Classifier Challenge, where the idea is to classify whether an object present in a radar image is an iceberg or a ship. I am currently trying to implement stochastic gradient descent to get a better idea for how mini batch training works, since the data set won't fit in memory after I perform data augmentation.

SKLearn has the following two libraries for implementing logistic regression, sklearn.LogisticRegression and sklearn.SGDClassifier. After experimenting for a while, I've noticed that the SGD classifier never performs better than the logistic regression classifier, and that there are some other behaviors in the results of the SGD classifier that make me question whether I've implemented it properly or whether it's innately buggy (I'm sure it's the former).

The following plot summarizes the performance of the SGDClassifier model versus LogisticRegression on the iceberg data set.

The color curves are SGDClassifier training runs with different numbers of mini-batches ranging from 50 to 1604 (the total number of training samples in the data set). Each was trained for 50 passes using SGDClassifier.partial_fit(). The straight horizontal lines represent the base accuracy when only 1s are predicted, the logistic regression score, and the SGD score when all 1604 training samples are fit over a single pass.

I'm having a hard time understanding the following:

1. Why does LogisticRegression so significantly outperform SGDClassifier even though both should be training in the same way on the same data?

2. Why does the performance so greatly decreases when passing over the data 50 times instead of just once? The latter makes sense to me if it's simply a matter of overfitting to the data set when making multiple runs over the data, though it would be good to understand how to prevent this with mini-batch training.

3. Why do the results so wildly vary with the mini-batch size in a non-monotonic way, even after 50 passes through the data? Since the data is shuffled in between runs I would expect any noisiness in the fits to be smoothed over after such a large number of iterations.

Here's the code used to generate the results:

# Load training date
df_train = pd.read_csv('train_data.csv', sep = ',', header = 0, index_col = 0)


# Set input names; 'band_i_j' is pixel j from input image i
img_size = 75*75
inputs = ['inc_angle'] + ['band_1_' + str(i) for i in range(img_size)] + ['band_2_' + str(i) for i in range(img_size)]

# Set output name
output = 'is_iceberg'

# Set na values to mean of column
inc_angle_mean = df_train['inc_angle'].mean()
df_train['inc_angle'].fillna(inc_angle_mean, inplace = True)

# Get num training samples
N_train = len(df_train)

# Calculate mean, max, min of each column; standardize inputs
train_means = df_train[inputs].mean()
train_maxes = df_train[inputs].max()
train_mins = df_train[inputs].min()
df_train[inputs] = (df_train[inputs] - train_means)/(train_maxes - train_mins)

# Set 80% of data to belong to training set; reserve the rest for validation set
train_indices = list(range(int(0.8*N_train)))
valid_indices = list(range(int(0.8*N_train), N_train))

# Base score
print('base score', len(df_train[df_train[output]==1])/N_train)

# SGD full-batch
model = sklearn.linear_model.SGDClassifier(loss= 'log', alpha = 1, tol = 0.00001, max_iter = 1000, shuffle = False, random_state = 0)
model.fit(df_train[inputs].iloc[train_indices], df_train[output].iloc[train_indices])
print('SGD full-batch', model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices]))

# Log reg
model = sklearn.linear_model.LogisticRegression(C = 1, tol = 0.00001, random_state = 0)
model.fit(df_train[inputs].iloc[train_indices], df_train[output].iloc[train_indices])
print('Log reg', model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices]))


# SGD mini-batch
num_passes = 50
batch_size = N_train
model = sklearn.linear_model.SGDClassifier(loss= 'log', alpha = 1, tol = 0.00001, shuffle = False, random_state = 0)

sgd_minibatch_scores = []
for i in range(num_passes):
    np.random.shuffle(train_indices)
    for j in range(int(len(train_indices)/batch_size + 1)):

        if j == int(len(train_indices)/batch_size+1):
            batch_train_indices = train_indices[j*batch_size:]
        else:
            batch_train_indices = train_indices[j*batch_size:(j+1)*batch_size]

        model.partial_fit(df_train[inputs].iloc[batch_train_indices], df_train[output].iloc[batch_train_indices], classes = [0,1])

    sgd_minibatch_scores.append(model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices]))

Answer 1

I found a related post here that suggests that a larger number of iterations are needed for convergence with sklearn.SGDClassifier(). After 3000 passes with sklearn.SGDClassifier() I was able to achieve around the same accuracy as sklearn.LogisticRegression(). I still find it strange that SGDLearn.fit() and LogisticRegression.fit() are not equivalent when training on the exact same samples and with the same arguments, but they must fundamentally train differently.