1
$\begingroup$

I am training a CNN to classify malware images from a dataset named Malimg. Before implementing the BatchNormalization layer, I was getting an accuracy of 95.57% (see below for the graph of loss/accuracy and validation loss/accuracy): accuracy loss

Epoch 1/10
6537/6537 [==============================] - 53s 8ms/step - loss: 1.7711 - accuracy: 0.4605 - val_loss: 1.0062 - val_accuracy: 0.6510
Epoch 2/10
6537/6537 [==============================] - 52s 8ms/step - loss: 0.8739 - accuracy: 0.7150 - val_loss: 0.4965 - val_accuracy: 0.8426
Epoch 3/10
6537/6537 [==============================] - 52s 8ms/step - loss: 0.5163 - accuracy: 0.8406 - val_loss: 0.3061 - val_accuracy: 0.9136
Epoch 4/10
6537/6537 [==============================] - 54s 8ms/step - loss: 0.3656 - accuracy: 0.8897 - val_loss: 0.1989 - val_accuracy: 0.9408
Epoch 5/10
6537/6537 [==============================] - 53s 8ms/step - loss: 0.3063 - accuracy: 0.9016 - val_loss: 0.1822 - val_accuracy: 0.9490
Epoch 6/10
6537/6537 [==============================] - 52s 8ms/step - loss: 0.2657 - accuracy: 0.9166 - val_loss: 0.1886 - val_accuracy: 0.9472
Epoch 7/10
6537/6537 [==============================] - 52s 8ms/step - loss: 0.2366 - accuracy: 0.9237 - val_loss: 0.1618 - val_accuracy: 0.9536
Epoch 8/10
6537/6537 [==============================] - 52s 8ms/step - loss: 0.2071 - accuracy: 0.9315 - val_loss: 0.1341 - val_accuracy: 0.9615
Epoch 9/10
6537/6537 [==============================] - 52s 8ms/step - loss: 0.2017 - accuracy: 0.9330 - val_loss: 0.1424 - val_accuracy: 0.9618
Epoch 10/10
6537/6537 [==============================] - 51s 8ms/step - loss: 0.1882 - accuracy: 0.9362 - val_loss: 0.1425 - val_accuracy: 0.9557
2802/2802 [==============================] - 9s 3ms/step

After implementing BatchNormalization, my results were very poor, and the model was all over the place (see below the results). Is there any reason for this? I am aware of what BatchNormalization does and that it stabilises the learning process and can help the neural network's performance, but I was advised to implement it.

Below is the code (of the neural network) and the results after implementing BatchNormalization:

num_classes = 25 #the amount of outputs in the output class
model-x = Sequential()
model-x.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(64,64,3)))
model-x.add(MaxPooling2D(pool_size=(2, 2)))
model-x.add(BatchNormalization())
model-x.add(Conv2D(16, (3, 3), activation='relu'))
model-x.add(MaxPooling2D(pool_size=(2, 2)))
model-x.add(Dropout(0.25))
model-x.add(Flatten())
model-x.add(Dense(128, activation='relu'))
model-x.add(Dropout(0.5))
model-x.add(Dense(50, activation='relu'))
model-x.add(Dense(num_classes, activation='softmax'))
model-x.compile(loss='categorical_crossentropy', optimizer = 'adam', metrics=['accuracy'])


y_train_new = np.argmax(y_train, axis=1)
class_weights = class_weight.compute_class_weight('balanced', np.unique(y_train_new), y_train_new)

history = model-x.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10,  class_weight=class_weights)
scores = model-x.evaluate(X_test, y_test)

model-x.summary()

accuracy - after batchnorm loss - after batchnorm

Updated results after suggestion accuracy - after suggestion loss- after suggestion

Code after suggestion

num_classes = 25 #the amount of outputs in the output class

model-x = Sequential()

model-x.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(64,64,3)))
model-x.add(BatchNormalization())
model-x.add(MaxPooling2D(pool_size=(2, 2)))
model-x.add(Conv2D(16, (3, 3), activation='relu'))
model-x.add(BatchNormalization())
model-x.add(MaxPooling2D(pool_size=(2, 2)))
model-x.add(Dropout(0.25))
model-x.add(Flatten())
model-x.add(Dense(128, activation='relu'))
model-x.add(Dropout(0.5))
model-x.add(Dense(50, activation='relu'))
model-x.add(Dense(num_classes, activation='softmax'))
model-x.compile(loss='categorical_crossentropy', optimizer = 'adam', metrics=['accuracy'])

y_train_new = np.argmax(y_train, axis=1)
class_weights = class_weight.compute_class_weight('balanced', np.unique(y_train_new), y_train_new)

history = model-x.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10,  class_weight=class_weights)
scores = model-x.evaluate(X_test, y_test)

model-x.summary()
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You are using the BatchNorm after a MaxPool. It is always applied after Convolution layers not Pooling layers. Hence, your network is acting like this.

Put BatchNorm after all Conv layers.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Hello. Thanks for your suggestion. I made changes (can be seen in the question now under the bold text). This produces very peculiar results too. $\endgroup$
    – Jack
    Commented Apr 20, 2021 at 18:34
  • $\begingroup$ Remove dropout after maxpool and keep dropout probability in between 0.1 and 0.3. $\endgroup$
    – Abhishek Verma
    Commented Apr 20, 2021 at 19:25
  • $\begingroup$ I will try and get back to you with the results. Is there by reason for removing dropout after maxpool and changing the probability? $\endgroup$
    – Jack
    Commented Apr 21, 2021 at 8:10
  • $\begingroup$ the results were not different and again very skewed even after your suggestion. $\endgroup$
    – Jack
    Commented Apr 21, 2021 at 11:30
  • $\begingroup$ You have very simple images. Start with a single conv layer with batch norm followed by a flatten and finally dense. No dropouts initially. Work with the kernel size, try 3 first then 5. Start from this, then, work incrementally. Also, read research papers on this dataset what they have done. See this dataset on paperswithcode. $\endgroup$
    – Abhishek Verma
    Commented Apr 21, 2021 at 11:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.