0
$\begingroup$

I am beginner in deep learning and I want to create a multi-input Convolutional Neural Network (CNN) model in Keras for Images Classification.

I am working to create a CNN model that takes two images and gives one output which is the class of the two images.

I have two datasets: type1 and type2, and each dataset contains the same classes, but the number of images in each class in the dataset type1 is higher than the number of images in each class in the dataset type2. The model should take one image from Type1 dataset and one image from Type2 dataset and then classify these images to one class (ClassA or ClassB or------).

The following is the structure of the datasets.

Type1 dataset
|Train
              |ClassA
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassB
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassC
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
              |ClassD
                             |image1
                             |image2
                             |image3
                             |image4
                            -----
       ----------------
|Validate
            -----------
|Test
           --------------

Type2 dataset
|Train
              |ClassA
                             |image1
                             |image2
                            -----
              |ClassB
                             |image1
                             |image2
                            -----
              |ClassC
                             |image1
                             |image2
                            -----
              |ClassD
                             |image1
                             |image2
                            -----
       ----------------
|Validate
            -----------
|Test
           --------------

The model is very similar to the model in this images, but it has more layers before flatten layer.

Multi-input

I created a custom generator that inputs two images (from type 1 and 2), and each image from type1 be paired with every images from type2 as long as these images belong to the same class (label).

The problem is when executing fit_generator I get infinite loop as the following:

  Found *** images belonging to 100 classes.

    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes.
    Found *** images belonging to 100 classes. ......
.................................................................

Here is my custom generator code:

input_imgen = ImageDataGenerator( 
                                  rotation_range=10,
                                  shear_range=0.2,
                                  zoom_range=0.1,
                                  width_shift_range=0.1,
                                  height_shift_range=0.1
                                  )



test_imgen = ImageDataGenerator()



def generate_generator_multiple(generator,dir1, dir2, batch_size, img_height,img_width):


    genX1 = generator.flow_from_directory(dir1,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)

    genX2 = generator.flow_from_directory(dir2,
                                          target_size = (img_height,img_width),
                                          class_mode = 'categorical',
                                          batch_size = batch_size,
                                          shuffle=False, 
                                          seed=7)
    while True:
      X2i = genX2.next() 
      Type1 = []
      Type2 = []
      image1 = []
      image2 = []

      while True:
        X1i = genX1.next() 
        for i in range(len(X2i[1])): #Type2
          for j in range(len(X1i[1])): #Type1
            if all(X2i[1][i]) == all(X1i[1][j]): # have same label
              image1.append(X1i[0][j]) # add image
              image1.append(X1i[1][j]) # add label
              image2.append(X2i[0][i]) # add image
              image2.append(X2i[1][i]) # add label
      Type1.append(image1)
      Type2.append(image2)
      yield [Type1 [0], Type2 [0]], Type2 [1]  #Yield both images and their mutual label


inputgenerator=generate_generator_multiple(generator=input_imgen,
                                           dir1=train_iris_data,
                                           dir2=train_face_data,
                                           batch_size=32,
                                           img_height=224,
                                           img_width=224)       

validgenerator=generate_generator_multiple(generator=test_imgen,
                                          dir1=valid_iris_data,
                                          dir2=valid_face_data,
                                          batch_size=32,
                                          img_height=224,
                                          img_width=224) 

testgenerator=generate_generator_multiple(generator=test_imgen,
                                          dir1=test_face_data,
                                          dir2=test_face_data,
                                          batch_size=32,
                                          img_height=224,
                                          img_width=224)


    # compile the model
    multi_model.compile(
            loss='categorical_crossentropy',
            optimizer=Adam(lr=0.0001),
            metrics=['accuracy']
        )


# train the model and save the history
history = multi_model.fit_generator(
inputgenerator,
steps_per_epoch=len(train_data) // batch_size,
epochs=10,
verbose=1,
validation_data=validgenerator,
validation_steps=len(valid_data) // batch_size,
use_multiprocessing=True,
shuffle=False
)

Could you please help me to solve this problem and create the custom generator??

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.