Multi scale CNN Network Python

I created a multi-scale CNN in python keras. The network architecture is similar to the diagram. Here, same image is fed to 3 CNN's with different architectures. The weights are NOT shared.

I coded the following multiscale CNN in keras which loosely resembles the architecture in the diagram. But I keep getting "Out of memory ERROR" even when the train_dir has 2 images. Would appreciate help...

#main CNN model - CNN1
main_model = Sequential()
main_model.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))

main_model.add(MaxPooling2D(pool_size=(2, 2))) # the main_model so far outputs 3D feature maps (height, width, features)

#lower features model - CNN2
lower_model1 = Sequential()
lower_model1.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))

#lower features model - CNN3
lower_model2 = Sequential()
lower_model2.add(Convolution2D(32, 3, 3, input_shape=(3, 224, 224)))

#merged model
merged_model = Merge([main_model, lower_model1, lower_model2], mode='concat')

final_model = Sequential()
final_model.compile(loss='binary_crossentropy', optimizer='rmsprop', metrics=['accuracy'])

print 'About to start training merged CNN'
train_datagen = ImageDataGenerator(rescale=1./255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
train_generator = train_datagen.flow_from_directory(train_data_dir, target_size=(224, 224), batch_size=32, class_mode='binary')

test_datagen = ImageDataGenerator(rescale=1./255)
test_generator = test_datagen.flow_from_directory(args.test_images, target_size=(224, 224), batch_size=32, class_mode='binary')

final_train_generator = zip(train_generator, train_generator, train_generator)
final_test_generator  = zip(test_generator, test_generator, test_generator)
final_model.fit_generator(final_train_generator, samples_per_epoch=nb_train_samples, nb_epoch=nb_epoch, validation_data=final_test_generator, nb_val_samples=nb_validation_samples)


UPDATE 1: Providing more system info

I am using Theano-0.9.0.dev5 | Keras-1.2.1 | Python 2.7.12 | OSX Sierra 10.12.3 (16D32) | Macbook Pro 16GB RAM | CPU mode

~/.keras/keras.json contents

{
"image_dim_ordering": "th",
"epsilon": 1e-07,
"floatx": "float64",
"backend": "theano"
}


Dont have .theanorc file

Please note individual CNN models are training fine. Only the merged code above causes issues.

UPDATE 2: on 27th January, 2017. Tried the following -

1. Reduced the no. of parameters of the CNN from 53 million to 100K. But still no use. The network eventually fails to train due to memory issues.

2. Reduced the batch size of images training to 8. The network training fails with the same reason.

No working solution at the time of writing this update...

• Some extra information would be useful for diagnosing the problem: version of keras, theano/tensorflow, cudnn and CUDA. Operating system. Configuration of keras (.keras/keras.conf) telling whether you are using GPU or CPU, etc. Configuration of theano/tensorflow; e.g. in .theanorc you can specify the memory management thresholds. Model of your GPU and its amount of RAM (if you are using a GPU). – noe Jan 24 '17 at 9:55
• @ncasas please find the update to the question with the data you requested. I also tried the same code in AWS GPU instances which resulted in the same issue aws.amazon.com/marketplace/pp/B01M0AXXQB – Srikar Appalaraju Jan 24 '17 at 10:50
• Have you tried with float32 instead of float64 in the keras configuration? – noe Jan 24 '17 at 11:08
• @ncasas yes just now changed to float32. Still same issue. code runs for 10minutes hogging RAM till it reaches around 40GB and is killed... – Srikar Appalaraju Jan 24 '17 at 11:56
• @ncasas I just tried final_model.summary() to see how many parameters used in the model. It shows 53 million parameters :o How is that possible? is my code wrong? The CNN's I defined are fairly simple... – Srikar Appalaraju Jan 24 '17 at 11:58

You could use the following function to determine how much memory your model requires:

def get_model_memory_usage(batch_size, model):
import numpy as np
from keras import backend as K

shapes_mem_count = 0
for l in model.layers:
single_layer_mem = 1
for s in l.output_shape:
if s is None:
continue
single_layer_mem *= s
shapes_mem_count += single_layer_mem

trainable_count = np.sum([K.count_params(p) for p in set(model.trainable_weights)])
non_trainable_count = np.sum([K.count_params(p) for p in set(model.non_trainable_weights)])

total_memory = 4.0*batch_size*(shapes_mem_count + trainable_count + non_trainable_count)
gbytes = np.round(total_memory / (1024.0 ** 3), 3)
return gbytes


I copied this from ZFTurbo's answer on this post.