I am working on a CNN for XRay image classification and I can't seem to be able to properly train it. I am trying to implement the following paper in Keras: https://arxiv.org/pdf/1801.09927.pdf

In short, the paper describes a 3 network architecture. Attention Guided CNN

The Global Branch is a ResNet or DenseNet (I used a DenseNet) pretrained on imagenet, then trained on the CheXNet dataset. After the training is done, the activations from the last convolutional layer are used to create heatmaps, which are then used to crop the images in the original dataset and create a new dataset.

The Local Branch is a ResNet or DenseNet (I used a DenseNet) with no pretraining. It is trained on the cropped dataset.

Finally, the fusion branch has as inputs the 2 global pooling layers of the Global and Local Branches.

I have managed to train the Global Branch (AUC: 0.77), generate the crops and train the Local Branch (AUC: 0.67). But when I try to train the Fusion Branch, the val_loss doesn't decrease: val_loss table

This is my code:

def load_model_from_json(models_folder):
    print(" --- Reading model from ", models_folder)
    with open(models_folder + 'model.json', 'r') as f:
        json = f.read()
    print("Read: ", models_folder + 'model.json')
    model = model_from_json(json)

    model.load_weights(models_folder + "model_weights.h5")
    print("Read: ", models_folder + "model_weights.h5")

    return model

global_branch_model = load_model_from_json(self.global_branch_path)

local_branch_model = load_model_from_json(self.local_branch_path)

for l in global_branch_model.layers:
    l.trainable = False
    l.name = 'global_'+l.name
for l in local_branch_model.layers:
    l.name = 'local_'+l.name
    l.trainable = False

global_pooling = global_branch_model.get_layer('global_global_average_pooling2d_1')

local_pooling = local_branch_model.get_layer('local_global_average_pooling2d_1')

merged = concatenate([global_pooling.output, local_pooling.output])
dense = Dense(512, activation='relu')(merged)
dropout = Dropout(self.hyperparameters.dropout)(dense)
out = Dense(1, activation='sigmoid')(dropout)

fusion_model = Model(inputs=[global_branch_model.input, local_branch_model.input], outputs=out)
loss_function = unweighted_binary_crossentropy

optimizer = AdamW(lr=5e-5)

fusion_model.compile(optimizer=optimizer, loss=loss_function)

            generator=FusionDataGenSequence(self.labels, self.partition['train'],
            # max_queue_size=32,
            # shuffle=False,
            # validation_steps=1

Can you please help me figure out what I did wrong? Thank you!


Maybe i'm missing something, but where is the connection between the global and local branches?

During the training of the local branch you used a dataset that was constructed from masked images. But now that branch should take as input the output of the 7x7x2048 conv layer from the global branch.

Maybe your training set constructed from coupled inputs for the 2 branches? I think you should add the implementation of your "FusionDataGenSequence".

What about your accuracy during the training? Did you try different learning rates?

  • $\begingroup$ The connection is here: global_pooling = global_branch_model.get_layer('global_global_average_pooling2d_1') local_pooling = local_branch_model.get_layer('local_global_average_pooling2d_1') After I generated the cropped dataset, I used that as input to the local branch. I did not measure my accuracy, only the val_loss and F1. I tried 5e-5 and 1e-7. $\endgroup$ Dec 24 '18 at 8:09

I found the problem. In the FusionDataGenSequence function I accidentally used a different normalization function for the images from the one used to train the branches.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.