I'm participating in the kaggle Iceberg Classifier Challenge, where the idea is to classify whether an object present in a radar image is an iceberg or a ship. I am currently trying to implement stochastic gradient descent to get a better idea for how mini batch training works, since the data set won't fit in memory after I perform data augmentation.
SKLearn has the following two libraries for implementing logistic regression,
sklearn.SGDClassifier. After experimenting for a while, I've noticed that the SGD classifier never performs better than the logistic regression classifier, and that there are some other behaviors in the results of the SGD classifier that make me question whether I've implemented it properly or whether it's innately buggy (I'm sure it's the former).
The following plot summarizes the performance of the
SGDClassifier model versus
LogisticRegression on the iceberg data set.
The color curves are
SGDClassifier training runs with different numbers of mini-batches ranging from 50 to 1604 (the total number of training samples in the data set). Each was trained for 50 passes using
SGDClassifier.partial_fit(). The straight horizontal lines represent the base accuracy when only 1s are predicted, the logistic regression score, and the SGD score when all 1604 training samples are fit over a single pass.
I'm having a hard time understanding the following:
LogisticRegressionso significantly outperform
SGDClassifiereven though both should be training in the same way on the same data?
Why does the performance so greatly decreases when passing over the data 50 times instead of just once? The latter makes sense to me if it's simply a matter of overfitting to the data set when making multiple runs over the data, though it would be good to understand how to prevent this with mini-batch training.
Why do the results so wildly vary with the mini-batch size in a non-monotonic way, even after 50 passes through the data? Since the data is shuffled in between runs I would expect any noisiness in the fits to be smoothed over after such a large number of iterations.
Here's the code used to generate the results:
# Load training date df_train = pd.read_csv('train_data.csv', sep = ',', header = 0, index_col = 0) # Set input names; 'band_i_j' is pixel j from input image i img_size = 75*75 inputs = ['inc_angle'] + ['band_1_' + str(i) for i in range(img_size)] + ['band_2_' + str(i) for i in range(img_size)] # Set output name output = 'is_iceberg' # Set na values to mean of column inc_angle_mean = df_train['inc_angle'].mean() df_train['inc_angle'].fillna(inc_angle_mean, inplace = True) # Get num training samples N_train = len(df_train) # Calculate mean, max, min of each column; standardize inputs train_means = df_train[inputs].mean() train_maxes = df_train[inputs].max() train_mins = df_train[inputs].min() df_train[inputs] = (df_train[inputs] - train_means)/(train_maxes - train_mins) # Set 80% of data to belong to training set; reserve the rest for validation set train_indices = list(range(int(0.8*N_train))) valid_indices = list(range(int(0.8*N_train), N_train)) # Base score print('base score', len(df_train[df_train[output]==1])/N_train) # SGD full-batch model = sklearn.linear_model.SGDClassifier(loss= 'log', alpha = 1, tol = 0.00001, max_iter = 1000, shuffle = False, random_state = 0) model.fit(df_train[inputs].iloc[train_indices], df_train[output].iloc[train_indices]) print('SGD full-batch', model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices])) # Log reg model = sklearn.linear_model.LogisticRegression(C = 1, tol = 0.00001, random_state = 0) model.fit(df_train[inputs].iloc[train_indices], df_train[output].iloc[train_indices]) print('Log reg', model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices])) # SGD mini-batch num_passes = 50 batch_size = N_train model = sklearn.linear_model.SGDClassifier(loss= 'log', alpha = 1, tol = 0.00001, shuffle = False, random_state = 0) sgd_minibatch_scores =  for i in range(num_passes): np.random.shuffle(train_indices) for j in range(int(len(train_indices)/batch_size + 1)): if j == int(len(train_indices)/batch_size+1): batch_train_indices = train_indices[j*batch_size:] else: batch_train_indices = train_indices[j*batch_size:(j+1)*batch_size] model.partial_fit(df_train[inputs].iloc[batch_train_indices], df_train[output].iloc[batch_train_indices], classes = [0,1]) sgd_minibatch_scores.append(model.score(df_train[inputs].iloc[valid_indices], df_train[output].iloc[valid_indices]))
C=alpha=1so that the regularization penalty should be the same. $\endgroup$