Multi-inputs Convolutional Neural Network takes different number of images

Question

I am using Keras to build a CNN model that takes two types of images as inputs (input1, input2) and produce one output. The model classifies the two inputs into a class, and the number of images in each type are different. (The number of images in the input1 is 1500 and in input2 is 4000).

This is my code:

in1 = Input(...)  
 x = Conv2D(...)(in1)
# rest of the model
 out1 = Dense(...)(x)  

 in2 = Input(...)  
x = Conv2D(...)(in2)
# rest of the model
out2 = Dense(...)(x)  

concatenated_layer = concatenate([out1, out2])  # merge the outputs of the two models
output_layer = Dense(no_classes, activation='softmax', name='prediction')(concatenated_layer)
modal= Model(inputs=[in1, in2], outputs=[output_layer])

Image generator code:

input_imgen = ImageDataGenerator(rescale = 1./255, 
                                   shear_range = 0.2, 
                                   zoom_range = 0.2,
                                   rotation_range=5.,
                                   horizontal_flip = True)

test_imgen = ImageDataGenerator()



def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):
    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)
    while True:
            X1i = genX1.next()
            X2i = genX2.next()
            yield [X1i[0], X2i[0]], X2i[1]  #Yield both images and their mutual label


inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_iris_data,
                                           dir2=train_face_data,
                                           batch_size=32,
                                           img_height=224,
                                           img_width=224)       

testgenerator=generate_generator_multiple(generator=test_imgen,
                                          dir1=valid_iris_data,
                                          dir2=valid_face_data,
                                          batch_size=1,
                                          img_height=224,
                                          img_width=224)              

My questions are:

How can I determine the value of steps_per_epoch and validation_steps if the number of the images in each type are different?

In general, How to deal with the multi input Convolutional Neural Network that takes different number of images?

Answer 1

You need to define what your training set is in this case. If a single input to your network is a pair of images $(a, b)$, and you only have 1500 examples of $b$, then you only have 1500 training examples.

If each image in $A$ can map to any arbitrary image in $B$, you can potentially re-use images in $B$ to create these input pairs. If that's not the case, which images in $A$ correspond to the image of interest in $B$?

Alternatively, if you don't want to (or it's not appropriate to) simply re-use the images, you could consider augmenting the smaller set by, for example, rotating, cropping, or otherwise manipulating the images in that dataset.

To re-initialize the generator, you would just execute this code again :

genX2 = generator.flow_from_directory(dir2, target_size = (img_height,img_width), class_mode = 'categorical', batch_size = batch_size, shuffle=False, seed=7)

When genX2.next() doesn't return anything. Alternatively, and more pythonically, you could use itertools.chain() to form a generator capable of returning the right number of observations:

n_times_through = math.ceil(4000/1500) 
genX2_new = itertools.chain(*([genX2] * n_times_through)