0
$\begingroup$

I try to test ResNet approach on cifar10 dataset with the following python code:

# load data
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32')/256
X_test = X_test.astype('float32')/256
y_train = keras.utils.to_categorical(y_train)
y_test = keras.utils.to_categorical(y_test)

# build a model
def res_unit(x):
    x_shortcut = x

    x = Conv2D(16, (1, 1), padding='same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)

    x = Conv2D(16, (3, 3), padding='same')(x)
    x = BatchNormalization()(x)
    x = Activation('relu')(x)

    x = Conv2D(32, (1, 1), padding='same')(x)
    x = BatchNormalization()(x)
    x = Add()([x_shortcut, x])

    x = Activation('relu')(x)

    return x


X_input = Input((X_train.shape[1:]))

X = Conv2D(32, (3, 3), padding='same')(X_input)
X = BatchNormalization()(X)
X = Activation('relu')(X)

X = res_unit(X)
X = res_unit(X)
X = res_unit(X)

X = Flatten()(X)
X = Dense(32)(X)
X = Activation('relu')(X)
X = Dense(10)(X)

X = Activation('softmax')(X)

model = Model(inputs=X_input, outputs=X)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

# run model
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=5)

which outputs the worst result for 10-class classification: 0.1 accuracy on train and validation.

But when I comment all lines with BatchNormalization, everything starts to be OK: I got 0.57 accuracy after first epoch and then it even rises up.

What is the problem with BatchNormalization in my case? Am I using it in a correct way?

$\endgroup$
0
$\begingroup$

I have played around with it and managed to get a working configuration. Necessary changes:

  • axis=1 for BatchNormalization. Normally, this should be used when data_format="channels_first", but it worked for me despite TensorFlow backend
  • Standardize the input:

    X_train = X_train * 2.0 - 1.0; X_test = X_test * 2.0 - 1.0

  • Avoid batchnorm after the last convolution in res_unit.

The working solution looks like this:

# build a model                                                                                                                                                                                                                                                       
def res_unit(x, prefix=''):                                                                                                                                                                                                                                           
    x_shortcut = x                                                                                                                                                                                                                                                    

    x = Conv2D(16, (1, 1), padding='same', name='{}/{}'.format(prefix, 'conv1'))(x)                                                                                                                                                                                   
    x = BatchNormalization(name='{}/{}/bn'.format(prefix, 'conv1'), axis=1)(x)                                                                                                                                                                                        
    x = Activation(activation='relu', name='{}/{}/relu'.format(prefix, 'conv1'))(x)                                                                                                                                                                                   

    x = Conv2D(16, (3, 3), padding='same', name='{}/{}'.format(prefix, 'conv2'))(x)                                                                                                                                                                                   
    x = BatchNormalization(name='{}/{}/bn'.format(prefix, 'conv2'), axis=1)(x)                                                                                                                                                                                        
    x = Activation(activation='relu', name='{}/{}/relu'.format(prefix, 'conv2'))(x)                                                                                                                                                                                   

    x = Conv2D(32, (1, 1), padding='same', name='{}/{}'.format(prefix, 'conv3'))(x)                                                                                                                                                                                   
    x = Add(name='{}/add'.format(prefix))([x_shortcut, x])                                                                                                                                                                                                            
    x = Activation('relu', name='{}/relu'.format(prefix))(x)                                                                                                                                                                                                          

    return x                                                                                                                                                                                                                                                          


def main():                                                                                                                                                                                                                                                           
    (X_train, y_train), (X_test, y_test) = load_data()                                                                                                                                                                                                                
    X_train = X_train.astype('float32') / 255                                                                                                                                                                                                                         
    X_test = X_test.astype('float32') / 255                                                                                                                                                                                                                           
    X_train = X_train * 2.0 - 1.0                                                                                                                                                                                                                                     
    X_test = X_test * 2.0 - 1.0                                                                                                                                                                                                                                       
    y_train = keras_utils.to_categorical(y_train)                                                                                                                                                                                                                     
    y_test = keras_utils.to_categorical(y_test)                                                                                                                                                                                                                       

    print(X_train.min())                                                                                                                                                                                                                                              
    print(X_train.max())                                                                                                                                                                                                                                              

    X_input = Input((X_train.shape[1:]))                                                                                                                                                                                                                              

    X = Conv2D(32, (3, 3), padding='same', name='conv0')(X_input)                                                                                                                                                                                                     
    X = BatchNormalization(name='conv0/bn', axis=1)(X)                                                                                                                                                                                                                
    X = Activation('relu', name='conv0/relu')(X)                                                                                                                                                                                                                      

    X = res_unit(X, prefix='block1')                                                                                                                                                                                                                                  
    X = res_unit(X, prefix='block2')                                                                                                                                                                                                                                  
    X = res_unit(X, prefix='block3')                                                                                                                                                                                                                                  

    X = Flatten()(X)                                                                                                                                                                                                                                                  
    X = Dense(32)(X)                                                                                                                                                                                                                                                  
    X = Activation('relu')(X)                                                                                                                                                                                                                                         
    X = Dense(10)(X)                                                                                                                                                                                                                                                  

    X = Activation('softmax', name='softmax')(X)                                                                                                                                                                                                                      

    model = Model(inputs=X_input, outputs=X)                                                                                                                                                                                                                          
    model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])

I've got 59.07% validation accuracy with it. However, sometimes training does not converge, it feels like it is very sensitive to the random init. Still don't know why making batchnorm work here is so hard...

$\endgroup$
6
  • $\begingroup$ According to the paper arxiv.org/pdf/1512.03385.pdf "add" operation should be applied before activation $\endgroup$ – Serge P. Jan 20 '19 at 17:33
  • $\begingroup$ According to the Keras ResNet example github.com/keras-team/keras/blob/master/examples/…, it should be applied to the same sort of activation, i.e. ReLU. $\endgroup$ – Dmytro Prylipko Jan 20 '19 at 21:41
  • $\begingroup$ The issue is not before or after activation. One term of addition is 'before' and the second one is 'after'. $\endgroup$ – Dmytro Prylipko Jan 20 '19 at 21:42
  • $\begingroup$ Most of ResNet implementations I saw do BatchNorm before Add followed by Activation. Anyway, swapping those lines in my code didn't helped. Removing batch normalization before Add() also has no effect $\endgroup$ – Serge P. Jan 21 '19 at 6:34
  • $\begingroup$ See the updated answer $\endgroup$ – Dmytro Prylipko Jan 21 '19 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.