# Multi-input Convolutional Neural Network for Images Classification

I am beginner in deep learning and I want to create a multi-input Convolutional Neural Network (CNN) model in Keras for Images Classification.

I am working to create a CNN model that takes two images and gives one output which is the class of the two images.

I have two datasets: type1 and type2, and each dataset contains the same classes, but the number of images in each class in the dataset type1 is higher than the number of images in each class in the dataset type2. The model should take one image from Type1 dataset and one image from Type2 dataset and then classify these images to one class (ClassA or ClassB or------).

The following is the structure of the datasets.

Type1 dataset
|Train
|ClassA
|image1
|image2
|image3
|image4
-----
|ClassB
|image1
|image2
|image3
|image4
-----
|ClassC
|image1
|image2
|image3
|image4
-----
|ClassD
|image1
|image2
|image3
|image4
-----
----------------
|Validate
-----------
|Test
--------------

Type2 dataset
|Train
|ClassA
|image1
|image2
-----
|ClassB
|image1
|image2
-----
|ClassC
|image1
|image2
-----
|ClassD
|image1
|image2
-----
----------------
|Validate
-----------
|Test
--------------


The model is very similar to the model in this images, but it has more layers before flatten layer.

I created a custom generator that inputs two images (from type 1 and 2), and each image from type1 be paired with every images from type2 as long as these images belong to the same class (label).

The problem is when executing fit_generator I get infinite loop as the following:

  Found *** images belonging to 100 classes.

Found *** images belonging to 100 classes.
Found *** images belonging to 100 classes.
Found *** images belonging to 100 classes.
Found *** images belonging to 100 classes.
Found *** images belonging to 100 classes.
Found *** images belonging to 100 classes.
Found *** images belonging to 100 classes.
Found *** images belonging to 100 classes.
Found *** images belonging to 100 classes.
Found *** images belonging to 100 classes.
Found *** images belonging to 100 classes. ......
.................................................................


Here is my custom generator code:

input_imgen = ImageDataGenerator(
rotation_range=10,
shear_range=0.2,
zoom_range=0.1,
width_shift_range=0.1,
height_shift_range=0.1
)

test_imgen = ImageDataGenerator()

def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):

genX1 = generator.flow_from_directory(dir1,
target_size = (img_height,img_width),
class_mode = 'categorical',
batch_size = batch_size,
shuffle=False,
seed=7)

genX2 = generator.flow_from_directory(dir2,
target_size = (img_height,img_width),
class_mode = 'categorical',
batch_size = batch_size,
shuffle=False,
seed=7)
while True:
X2i = genX2.next()
Type1 = []
Type2 = []
image1 = []
image2 = []

while True:
X1i = genX1.next()
for i in range(len(X2i[1])): #Type2
for j in range(len(X1i[1])): #Type1
if all(X2i[1][i]) == all(X1i[1][j]): # have same label
Type1.append(image1)
Type2.append(image2)
yield [Type1 [0], Type2 [0]], Type2 [1]  #Yield both images and their mutual label

inputgenerator=generate_generator_multiple(generator=input_imgen,
dir1=train_iris_data,
dir2=train_face_data,
batch_size=32,
img_height=224,
img_width=224)

validgenerator=generate_generator_multiple(generator=test_imgen,
dir1=valid_iris_data,
dir2=valid_face_data,
batch_size=32,
img_height=224,
img_width=224)

testgenerator=generate_generator_multiple(generator=test_imgen,
dir1=test_face_data,
dir2=test_face_data,
batch_size=32,
img_height=224,
img_width=224)

# compile the model
multi_model.compile(
loss='categorical_crossentropy',
metrics=['accuracy']
)

# train the model and save the history
history = multi_model.fit_generator(
inputgenerator,
steps_per_epoch=len(train_data) // batch_size,
epochs=10,
verbose=1,
validation_data=validgenerator,
validation_steps=len(valid_data) // batch_size,
use_multiprocessing=True,
shuffle=False
)