0
$\begingroup$

I am using cifar10 dataset and below is the code that I am using. I think that the model is regularized but after around 0.70 of validation accuracy, it plateaus. Following are the graphs of loss and accuracies. I can only use CNN, this is part of assignment.

Even when i increase the regularization parameters, the validation accuracy is not crossing 0.70.

I would like some advices on how to increase this validation accuracy. I have been facing this problem in 3 more cases.

Loss vs epochs and Accuracies vs epochs for both training and validation dataset

(x_train, y_train), (x_val, y_val) = cifar10.load_data()

# =========================================================
# One hot encoding the labels
# =========================================================

y_train_ohe = tf.keras.utils.to_categorical(y_train, num_classes=10, dtype='float32')

y_val_ohe = tf.keras.utils.to_categorical(y_val, num_classes=10, dtype='float32')
# =========================================================
# SCaling the inputs
# =========================================================

from sklearn.model_selection import train_test_split

# x_train_scaled = x_train / 255.
x_val_scaled = x_val / 255.

y_train_upsam = np.concatenate((y_train_ohe, y_train_ohe))

def return_datasets():
  trainX, testX, trainY, testY = train_test_split(x_train, y_train_ohe, train_size=0.7, stratify= y_train_ohe)

  train = tf.data.Dataset.from_tensor_slices((trainX/255., trainY)).batch(BATCH_SIZE).prefetch(AUTOTUNE)
  test = tf.data.Dataset.from_tensor_slices((testX/255, testY)).batch(BATCH_SIZE).prefetch(AUTOTUNE)
  return train, test
# ==============================================================
# Building model
# this is not alexnet, i was previously using it but then
# changed the architecture and did not change the variable names
# ==============================================================

# clearing any previous sessions
tf.keras.backend.clear_session()

# parameters
# lr = 0.001                  # default
opt = tf.keras.optimizers.Adam(0.001, decay=1e-8)
epo = 50      # number of epochs
loss_fn = tf.keras.losses.CategoricalCrossentropy()
BATCH_SIZE = 32
reg = tf.keras.regularizers.L1L2(l1=0.0025, l2=0.0025)
# initializer = tf.keras.initializers.HeNormal()
# cb = tf.keras.callbacks.EarlyStopping(patience=50, restore_best_weights=True)

# def scheduler(epoch, lr):
#   if epoch % 5 == 0:
#     lr *=0.9
#   return lr

# making a model
aNet = tf.keras.models.Sequential(
    [
     Conv2D(128, kernel_size=3, activation='relu', input_shape=(x_train.shape[1:]),),
     Conv2D(128, kernel_size=3, padding='same', activation='relu',),
     MaxPooling2D(),

     Conv2D(128, kernel_size=3,activation='relu',),
     Conv2D(128, kernel_size=3, padding='same', activation='relu',),
     MaxPooling2D(),

     Conv2D(128, kernel_size=3, activation='relu',),
     Conv2D(128, kernel_size=3, padding='same', activation='relu',),
     MaxPooling2D(),

     Flatten(),
     
     Dense(256, activation='relu', kernel_regularizer=reg),
     Dropout(0.5),
     Dense(256, activation='relu', kernel_regularizer=reg),
     Dropout(0.5),

     Dense(10, activation='softmax'),
    ]
)


aNet.compile(optimizer = opt,
             loss = loss_fn,
             metrics = ['accuracy'])

train, test = return_datasets()

print("Fitting the model")
history_anet_1 = aNet.fit(train, 
                            validation_data = test,
                            epochs = epo,
                            batch_size = BATCH_SIZE,
                            verbose=2,)
                          # callbacks=[tf.keras.callbacks.LearningRateScheduler(scheduler)])

print("\nModel Fitting finished")

print(aNet.summary())
$\endgroup$

1 Answer 1

2
$\begingroup$

Classic case of overfitting. Your training loss keeps decreasing but your validation increases roughly after the 10th epoch. I'd first consider changing the down-sizing of some of your convolutional and fully connected layers such that, they are not all the same size. Too many neurons can cause the network to "memorize" the training input, making it harder the generalize to unseen data.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.