# Multi-inputs Convolutional Neural Network takes different number of images

I am using Keras to build a CNN model that takes two types of images as inputs (input1, input2) and produce one output. The model classifies the two inputs into a class, and the number of images in each type are different. (The number of images in the input1 is 1500 and in input2 is 4000).

This is my code:

in1 = Input(...)
x = Conv2D(...)(in1)
# rest of the model
out1 = Dense(...)(x)

in2 = Input(...)
x = Conv2D(...)(in2)
# rest of the model
out2 = Dense(...)(x)

concatenated_layer = concatenate([out1, out2])  # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(concatenated_layer)
modal= Model(inputs=[in1, in2], outputs=[output_layer])


Image generator code:

input_imgen = ImageDataGenerator(rescale = 1./255,
shear_range = 0.2,
zoom_range = 0.2,
rotation_range=5.,
horizontal_flip = True)

test_imgen = ImageDataGenerator()

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
genX1 = generator.flow_from_directory(dir1,
target_size = (img_height,img_width),
class_mode = 'categorical',
batch_size = batch_size,
shuffle=False,
seed=7)

genX2 = generator.flow_from_directory(dir2,
target_size = (img_height,img_width),
class_mode = 'categorical',
batch_size = batch_size,
shuffle=False,
seed=7)
while True:
X1i = genX1.next()
X2i = genX2.next()
yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=input_imgen,
dir1=train_iris_data,
dir2=train_face_data,
batch_size=32,
img_height=224,
img_width=224)

testgenerator=generate_generator_multiple(generator=test_imgen,
dir1=valid_iris_data,
dir2=valid_face_data,
batch_size=1,
img_height=224,
img_width=224)


My questions are:

How can I determine the value of steps_per_epoch and validation_steps if the number of the images in each type are different?

In general, How to deal with the multi input Convolutional Neural Network that takes different number of images?

You need to define what your training set is in this case. If a single input to your network is a pair of images $$(a, b)$$, and you only have 1500 examples of $$b$$, then you only have 1500 training examples.

If each image in $$A$$ can map to any arbitrary image in $$B$$, you can potentially re-use images in $$B$$ to create these input pairs. If that's not the case, which images in $$A$$ correspond to the image of interest in $$B$$?

Alternatively, if you don't want to (or it's not appropriate to) simply re-use the images, you could consider augmenting the smaller set by, for example, rotating, cropping, or otherwise manipulating the images in that dataset.

To re-initialize the generator, you would just execute this code again :

genX2 = generator.flow_from_directory(dir2, target_size = (img_height,img_width), class_mode = 'categorical', batch_size = batch_size, shuffle=False, seed=7)


When genX2.next() doesn't return anything. Alternatively, and more pythonically, you could use itertools.chain() to form a generator capable of returning the right number of observations:

n_times_through = math.ceil(4000/1500)
genX2_new = itertools.chain(*([genX2] * n_times_through)

• My case is different images from the first input can map to different images in B. I mean different images in A and different images in B have the same class. Also, I am using data augmentation for the two inputs. I think I have to re-use images in B, right ?? How can I do that? – Noran Nov 2 '18 at 15:52
• you could, in generate_generator_multiple, simply re-initialize genX2 when genX2.next() stops generating an image from the iterator. – Thomas Cleberg Nov 2 '18 at 16:20
• @Thomas Cleberg How to re-initialize genX2? Could you explain? or give an example? please. – N.IT Nov 4 '18 at 12:23
• Can you explain how can I re-initialize genX2? Also, during training I want to paired images (a,b) from the same class @ThomasCleberg – Noran Nov 7 '18 at 18:28
• Added content on methods to re-use. – Thomas Cleberg Nov 8 '18 at 17:13