0
$\begingroup$

What may cause the accuracy reduction when using the tf.keras regularizer at layers in CNN in the symptom? The example is simple but it happens with more complex CNN causing no improvement during the training.

I was thinking adding regularizer is a good way, but apparently it can harm the model training. Please advise what can cause it and if there is rules to follow on how to configure and use the layer regularizers.

Symptom

Without regularizer

250/1250 [...  - loss: 1.7895 - accuracy: 0.4002 - val_loss: 1.4933 - val_accuracy: 0.4894
Epoch 2/3
1250/1250 [... - loss: 1.4432 - accuracy: 0.4811 - val_loss: 1.6485 - val_accuracy: 0.4870
Epoch 3/3
1250/1250 [... - loss: 1.3627 - accuracy: 0.5133 - val_loss: 1.3166 - val_accuracy: 0.5657

With kernel regulizer

Epoch 1/3
1250/1250 [... - loss: 3.4165 - accuracy: 0.3751 - val_loss: 2.2260 - val_accuracy: 0.4072
Epoch 2/3
1250/1250 [... - loss: 2.1345 - accuracy: 0.4172 - val_loss: 2.0945 - val_accuracy: 0.4376
Epoch 3/3
1250/1250 [... - loss: 2.1065 - accuracy: 0.4257 - val_loss: 2.0685 - val_accuracy: 0.4231

Code

import tensorflow as tf
from keras.layers import (
    Conv2D,
    MaxPooling2D,
    BatchNormalization,
    Dense,
    Flatten,
    Dropout,
)
from keras.models import (
    Sequential
)
from keras.optimizers import (
    Adam
)
from keras.preprocessing.image import (
    ImageDataGenerator
)
from sklearn.model_selection import (
    train_test_split
)


NUM_CLASSES = 10
BATCH_SIZE = 32
EPOCHS = 3

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.cifar10.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0
x_train, x_validation, y_train, y_validation = train_test_split(
    x_train, y_train, test_size=0.2, random_state=42
)

y_train = tf.keras.utils.to_categorical(y_train, NUM_CLASSES)
y_validation = tf.keras.utils.to_categorical(y_validation, NUM_CLASSES)

# set up image augmentation
datagen = ImageDataGenerator(
    rotation_range=15,
    horizontal_flip=True,
    width_shift_range=0.1,
    height_shift_range=0.1
)

def build(reg):
    model = Sequential()
    model.add(Conv2D(32, (3, 3), activation='relu', kernel_regularizer=reg, input_shape=(32, 32, 3),padding='same'))
    model.add(BatchNormalization(axis=-1))
    model.add(MaxPooling2D(pool_size=(2, 2))) 
    model.add(Flatten())
    model.add(Dense(512, activation='relu',kernel_regularizer=reg))
    model.add(BatchNormalization())
    model.add(Dropout(0.5))
    model.add(Dense(NUM_CLASSES, activation='softmax'))
    
    model.compile(
        loss='categorical_crossentropy', 
        metrics=['accuracy'],
        optimizer=Adam(learning_rate=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08)
    )
    return model

# --------------------------------------------------------------------------------
# Model layer without kernel regularizer
# --------------------------------------------------------------------------------
model = build(reg=None)
model.summary()

history = model.fit(
    x=datagen.flow(x_train, y_train, batch_size=BATCH_SIZE),
    steps_per_epoch = len(x_train) / BATCH_SIZE, 
    epochs=EPOCHS, 
    validation_data=(x_validation, y_validation)
)

# --------------------------------------------------------------------------------
# Model layer without kernel regularizer
# --------------------------------------------------------------------------------
model = build(reg=tf.keras.regularizers.L2(l2=0.01))
model.summary()

history2 = model.fit(
    x=datagen.flow(x_train, y_train, batch_size=BATCH_SIZE),
    steps_per_epoch = len(x_train) / BATCH_SIZE, 
    epochs=EPOCHS, 
    validation_data=(x_validation, y_validation)
)

Model summaries

Without regularizer

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d (Conv2D)             (None, 32, 32, 32)        896       
                                                                 
 batch_normalization (Batch  (None, 32, 32, 32)        128       
 Normalization)                                                  
                                                                 
 max_pooling2d (MaxPooling2  (None, 16, 16, 32)        0         
 D)                                                              
                                                                 
 flatten (Flatten)           (None, 8192)              0         
                                                                 
 dense (Dense)               (None, 512)               4194816   
                                                                 
 batch_normalization_1 (Bat  (None, 512)               2048      
 chNormalization)                                                
                                                                 
 dropout (Dropout)           (None, 512)               0         
                                                                 
 dense_1 (Dense)             (None, 10)                5130      
                                                                 
=================================================================
Total params: 4203018 (16.03 MB)
Trainable params: 4201930 (16.03 MB)
Non-trainable params: 1088 (4.25 KB)
_________________________________________________________________

With regularizer

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 conv2d_1 (Conv2D)           (None, 32, 32, 32)        896       
                                                                 
 batch_normalization_2 (Bat  (None, 32, 32, 32)        128       
 chNormalization)                                                
                                                                 
 max_pooling2d_1 (MaxPoolin  (None, 16, 16, 32)        0         
 g2D)                                                            
                                                                 
 flatten_1 (Flatten)         (None, 8192)              0         
                                                                 
 dense_2 (Dense)             (None, 512)               4194816   
                                                                 
 batch_normalization_3 (Bat  (None, 512)               2048      
 chNormalization)                                                
                                                                 
 dropout_1 (Dropout)         (None, 512)               0         
                                                                 
 dense_3 (Dense)             (None, 10)                5130      
                                                                 
=================================================================
Total params: 4203018 (16.03 MB)
Trainable params: 4201930 (16.03 MB)
Non-trainable params: 1088 (4.25 KB)
_________________________________________________________________

Environment

TensorFlow version: 2.14.1
Keras version: 2.14.0
Python 3.10.12
Ubuntu 22.04 LTS

Note

For more layers, the training does not improve with the regularizers where the val_accuracy stays around 0.01.

_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 image (InputLayer)          [(None, 32, 32, 3)]       0         
                                                                 
 conv01 (Conv2D)             (None, 32, 32, 32)        896       
                                                                 
 bn01 (BatchNormalization)   (None, 32, 32, 32)        128       
                                                                 
 conv01_2 (Conv2D)           (None, 32, 32, 32)        9248      
                                                                 
 bn01_2 (BatchNormalization  (None, 32, 32, 32)        128       
 )                                                               
                                                                 
 maxpool01 (MaxPooling2D)    (None, 16, 16, 32)        0         
                                                                 
 drop01 (Dropout)            (None, 16, 16, 32)        0         
                                                                 
 conv02 (Conv2D)             (None, 16, 16, 64)        18496     
                                                                 
 bn02 (BatchNormalization)   (None, 16, 16, 64)        256       
                                                                 
 conv02_2 (Conv2D)           (None, 16, 16, 64)        36928     
                                                                 
 bn02_2 (BatchNormalization  (None, 16, 16, 64)        256       
 )                                                               
                                                                 
 maxpool02 (MaxPooling2D)    (None, 8, 8, 64)          0         
                                                                 
 drop02 (Dropout)            (None, 8, 8, 64)          0         
                                                                 
 conv03_1 (Conv2D)           (None, 8, 8, 128)         73856     
                                                                 
 bn03 (BatchNormalization)   (None, 8, 8, 128)         512       
                                                                 
 conv03_2 (Conv2D)           (None, 8, 8, 128)         147584    
                                                                 
 bn03_2 (BatchNormalization  (None, 8, 8, 128)         512       
 )                                                               
                                                                 
 maxpool03 (MaxPooling2D)    (None, 4, 4, 128)         0         
                                                                 
 drop03 (Dropout)            (None, 4, 4, 128)         0         
                                                                 
 flat (Flatten)              (None, 2048)              0         
                                                                 
 full01 (Dense)              (None, 512)               1049088   
                                                                 
 bn (BatchNormalization)     (None, 512)               2048      
                                                                 
 drop (Dropout)              (None, 512)               0         
                                                                 
 classification (Dense)      (None, 10)                5130      
                                                                 
=================================================================
Total params: 1345066 (5.13 MB)
Trainable params: 1343146 (5.12 MB)
Non-trainable params: 1920 (7.50 KB)
_________________________________________________________________
```
Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

In general, I stumbled across voices in literature that we shouldn't use dropout with such a big parameter for shallow networks, as it can violate their capabilities. Example:

Piotrowski, A. P., Napiorkowski, J. J., & Piotrowska, A. E. (2020). Impact of deep learning-based dropout on shallow neural networks applied to stream temperature modelling. Earth-Science Reviews, 201, 103076.

Also, from my experience, your network appears too shallow. The additional aftermath of that is that your first Dense layer is unnecessarily big and it can even hinder the potential of your model. What you can try from my point of view:

  • decrease Dropout's parameter
  • add convolutional layer(s) before the Dense layer
  • add Dropout also after convolutional layer
  • normalization first, activation afterwards - as a plus
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the answer and reference. My apology that I should have put background. The original network is deeper. However, observed the model training accuracy was not improving. By trimming down to very shallow one, found kernel regularizer causing the issue (by removing them, accuracy started improving). Hence, trying to understand what the mechanism of kernel regulazer can cause the issue. $\endgroup$
    – mon
    Commented Nov 25, 2023 at 2:09
  • $\begingroup$ Regarding normalization first, activation afterwards, kindly provide why the order of normalization first would be better? $\endgroup$
    – mon
    Commented Nov 25, 2023 at 2:11
  • $\begingroup$ Thank you for this question, it made me review my knowledge and I found it more complex than I thought. I suggested using normalisation first because I saw it many times in literature but seldom in practice. My intuition is that if you normalised after activation, you would often cut the entire or none distribution of data in many cases. Normalisation allows to move of data to be cut partially by normalisation. I always use it on my own. But I stumbled across a paper that made the issue ambiguous: openreview.net/forum?id=FLr9RRqbwB-. It seems it depends on the type of activation. $\endgroup$
    – Tomasz Witkowski
    Commented Nov 25, 2023 at 7:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.