I created a multi-scale CNN in python keras. The network architecture is similar to the diagram. Here, same image is fed to 3 CNN's with different architectures. The weights are NOT shared.

enter image description here

I coded the following multiscale CNN in keras which loosely resembles the architecture in the diagram. But I keep getting "Out of memory ERROR" even when the train_dir has 2 images. Would appreciate help...

#main CNN model - CNN1
main_model = Sequential()
main_model.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))
main_model.add(MaxPooling2D(pool_size=(2, 2)))

main_model.add(Convolution2D(32, 3, 3))
main_model.add(MaxPooling2D(pool_size=(2, 2)))

main_model.add(Convolution2D(64, 3, 3))
main_model.add(MaxPooling2D(pool_size=(2, 2))) # the main_model so far outputs 3D feature maps (height, width, features)


#lower features model - CNN2
lower_model1 = Sequential()
lower_model1.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))
lower_model1.add(MaxPooling2D(pool_size=(2, 2)))

#lower features model - CNN3
lower_model2 = Sequential()
lower_model2.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))
lower_model2.add(MaxPooling2D(pool_size=(2, 2)))

#merged model
merged_model = Merge([main_model, lower_model1, lower_model2], mode='concat')

final_model = Sequential()                     
final_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

print 'About to start training merged CNN'
train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
train_generator = train_datagen.flow_from_directory(train_data_dir, target_size=(224, 224), batch_size=32, class_mode='binary')

test_datagen = ImageDataGenerator(rescale=1./255)
test_generator = test_datagen.flow_from_directory(args.test_images, target_size=(224, 224), batch_size=32, class_mode='binary')

final_train_generator = zip(train_generator, train_generator, train_generator)
final_test_generator  = zip(test_generator, test_generator, test_generator)
final_model.fit_generator(final_train_generator, samples_per_epoch=nb_train_samples, nb_epoch=nb_epoch, validation_data=final_test_generator, nb_val_samples=nb_validation_samples)

UPDATE 1: Providing more system info

I am using Theano-0.9.0.dev5 | Keras-1.2.1 | Python 2.7.12 | OSX Sierra 10.12.3 (16D32) | Macbook Pro 16GB RAM | CPU mode

~/.keras/keras.json contents

    "image_dim_ordering": "th",
    "epsilon": 1e-07,
    "floatx": "float64",
    "backend": "theano"

Dont have .theanorc file

Please note individual CNN models are training fine. Only the merged code above causes issues.

UPDATE 2: on 27th January, 2017. Tried the following -

  1. Reduced the no. of parameters of the CNN from 53 million to 100K. But still no use. The network eventually fails to train due to memory issues.

  2. Reduced the batch size of images training to 8. The network training fails with the same reason.

No working solution at the time of writing this update...

  • 1
    $\begingroup$ Some extra information would be useful for diagnosing the problem: version of keras, theano/tensorflow, cudnn and CUDA. Operating system. Configuration of keras (.keras/keras.conf) telling whether you are using GPU or CPU, etc. Configuration of theano/tensorflow; e.g. in .theanorc you can specify the memory management thresholds. Model of your GPU and its amount of RAM (if you are using a GPU). $\endgroup$
    – noe
    Jan 24, 2017 at 9:55
  • $\begingroup$ @ncasas please find the update to the question with the data you requested. I also tried the same code in AWS GPU instances which resulted in the same issue aws.amazon.com/marketplace/pp/B01M0AXXQB $\endgroup$ Jan 24, 2017 at 10:50
  • 1
    $\begingroup$ Have you tried with float32 instead of float64 in the keras configuration? $\endgroup$
    – noe
    Jan 24, 2017 at 11:08
  • $\begingroup$ @ncasas yes just now changed to float32. Still same issue. code runs for 10minutes hogging RAM till it reaches around 40GB and is killed... $\endgroup$ Jan 24, 2017 at 11:56
  • $\begingroup$ @ncasas I just tried final_model.summary() to see how many parameters used in the model. It shows 53 million parameters :o How is that possible? is my code wrong? The CNN's I defined are fairly simple... $\endgroup$ Jan 24, 2017 at 11:58

1 Answer 1


You could use the following function to determine how much memory your model requires:

def get_model_memory_usage(batch_size, model):
    import numpy as np
    from keras import backend as K

    shapes_mem_count = 0
    for l in model.layers:
        single_layer_mem = 1
        for s in l.output_shape:
            if s is None:
            single_layer_mem *= s
        shapes_mem_count += single_layer_mem

    trainable_count = np.sum([K.count_params(p) for p in set(model.trainable_weights)])
    non_trainable_count = np.sum([K.count_params(p) for p in set(model.non_trainable_weights)])

    total_memory = 4.0*batch_size*(shapes_mem_count + trainable_count + non_trainable_count)
    gbytes = np.round(total_memory / (1024.0 ** 3), 3)
    return gbytes

I copied this from ZFTurbo's answer on this post.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.