6
$\begingroup$

I'm participating in the kaggle Iceberg Classifier Challenge, where the idea is to classify whether an object present in a radar image is an iceberg or a ship. I am currently trying to implement stochastic gradient descent to get a better idea for how mini batch training works, since the data set won't fit in memory after I perform data augmentation.

SKLearn has the following two libraries for implementing logistic regression, sklearn.LogisticRegression and sklearn.SGDClassifier. After experimenting for a while, I've noticed that the SGD classifier never performs better than the logistic regression classifier, and that there are some other behaviors in the results of the SGD classifier that make me question whether I've implemented it properly or whether it's innately buggy (I'm sure it's the former).

The following plot summarizes the performance of the SGDClassifier model versus LogisticRegression on the iceberg data set.

sgd-vs-logreg

The color curves are SGDClassifier training runs with different numbers of mini-batches ranging from 50 to 1604 (the total number of training samples in the data set). Each was trained for 50 passes using SGDClassifier.partial_fit(). The straight horizontal lines represent the base accuracy when only 1s are predicted, the logistic regression score, and the SGD score when all 1604 training samples are fit over a single pass.

I'm having a hard time understanding the following:

  1. Why does LogisticRegression so significantly outperform SGDClassifier even though both should be training in the same way on the same data?

  2. Why does the performance so greatly decreases when passing over the data 50 times instead of just once? The latter makes sense to me if it's simply a matter of overfitting to the data set when making multiple runs over the data, though it would be good to understand how to prevent this with mini-batch training.

  3. Why do the results so wildly vary with the mini-batch size in a non-monotonic way, even after 50 passes through the data? Since the data is shuffled in between runs I would expect any noisiness in the fits to be smoothed over after such a large number of iterations.

Here's the code used to generate the results:

# Load training date
df_train = pd.read_csv('train_data.csv', sep = ',', header = 0, index_col = 0)


# Set input names; 'band_i_j' is pixel j from input image i
img_size = 75*75
inputs = ['inc_angle'] + ['band_1_' + str(i) for i in range(img_size)] + ['band_2_' + str(i) for i in range(img_size)]

# Set output name
output = 'is_iceberg'

# Set na values to mean of column
inc_angle_mean = df_train['inc_angle'].mean()
df_train['inc_angle'].fillna(inc_angle_mean, inplace = True)

# Get num training samples
N_train = len(df_train)

# Calculate mean, max, min of each column; standardize inputs
train_means = df_train[inputs].mean()
train_maxes = df_train[inputs].max()
train_mins = df_train[inputs].min()
df_train[inputs] = (df_train[inputs] - train_means)/(train_maxes - train_mins)

# Set 80% of data to belong to training set; reserve the rest for validation set
train_indices = list(range(int(0.8*N_train)))
valid_indices = list(range(int(0.8*N_train), N_train))

# Base score
print('base score', len(df_train[df_train[output]==1])/N_train)

# SGD full-batch
model = sklearn.linear_model.SGDClassifier(loss= 'log', alpha = 1, tol = 0.00001, max_iter = 1000, shuffle = False, random_state = 0)
model.fit(df_train[inputs].iloc[train_indices], df_train[output].iloc[train_indices])
print('SGD full-batch', model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices]))

# Log reg
model = sklearn.linear_model.LogisticRegression(C = 1, tol = 0.00001, random_state = 0)
model.fit(df_train[inputs].iloc[train_indices], df_train[output].iloc[train_indices])
print('Log reg', model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices]))


# SGD mini-batch
num_passes = 50
batch_size = N_train
model = sklearn.linear_model.SGDClassifier(loss= 'log', alpha = 1, tol = 0.00001, shuffle = False, random_state = 0)

sgd_minibatch_scores = []
for i in range(num_passes):
    np.random.shuffle(train_indices)
    for j in range(int(len(train_indices)/batch_size + 1)):

        if j == int(len(train_indices)/batch_size+1):
            batch_train_indices = train_indices[j*batch_size:]
        else:
            batch_train_indices = train_indices[j*batch_size:(j+1)*batch_size]

        model.partial_fit(df_train[inputs].iloc[batch_train_indices], df_train[output].iloc[batch_train_indices], classes = [0,1])

    sgd_minibatch_scores.append(model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices]))
$\endgroup$
  • $\begingroup$ Is the input scaled? $\endgroup$ – El Burro Nov 30 '17 at 10:58
  • $\begingroup$ Yes, but I originally missed it when I copied the code into this post. The original post has been updated. $\endgroup$ – PrestonH Nov 30 '17 at 17:00
  • $\begingroup$ Try playing around with different values of alpha. Its meaning is different from C. It might have quite an impact because the number of features is too high compared to the sample size. $\endgroup$ – Michael M Dec 1 '17 at 8:10
  • $\begingroup$ Are you sure? From the documentation it states that C is the inverse of the regularization strength and alpha is 'Constant that multiplies the regularization term'. I think that implies alpha = 1/C, which is consistent with other notation I've seen elsewhere. In my case I set C=alpha=1 so that the regularization penalty should be the same. $\endgroup$ – PrestonH Dec 1 '17 at 18:57
  • $\begingroup$ LogisticRegression uses LIBLINEAR optimization if you are not mentioning any optimization method(solver) as an argument which is different than basic SGD which is used in SGDClassifier so you may get good results on one of them, that totally depends on how we converged using that optimization technique and data we have. $\endgroup$ – Uday Oct 15 '19 at 10:44
1
$\begingroup$

I found a related post here that suggests that a larger number of iterations are needed for convergence with sklearn.SGDClassifier(). After 3000 passes with sklearn.SGDClassifier() I was able to achieve around the same accuracy as sklearn.LogisticRegression(). I still find it strange that SGDLearn.fit() and LogisticRegression.fit() are not equivalent when training on the exact same samples and with the same arguments, but they must fundamentally train differently.

sgd-vs-logreg

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.