Keras Attention Guided CNN problem

I am working on a CNN for XRay image classification and I can't seem to be able to properly train it. I am trying to implement the following paper in Keras: https://arxiv.org/pdf/1801.09927.pdf

In short, the paper describes a 3 network architecture.

The Global Branch is a ResNet or DenseNet (I used a DenseNet) pretrained on imagenet, then trained on the CheXNet dataset. After the training is done, the activations from the last convolutional layer are used to create heatmaps, which are then used to crop the images in the original dataset and create a new dataset.

The Local Branch is a ResNet or DenseNet (I used a DenseNet) with no pretraining. It is trained on the cropped dataset.

Finally, the fusion branch has as inputs the 2 global pooling layers of the Global and Local Branches.

I have managed to train the Global Branch (AUC: 0.77), generate the crops and train the Local Branch (AUC: 0.67). But when I try to train the Fusion Branch, the val_loss doesn't decrease:

This is my code:

def load_model_from_json(models_folder):
print(" --- Reading model from ", models_folder)
with open(models_folder + 'model.json', 'r') as f:
model = model_from_json(json)

return model

for l in global_branch_model.layers:
l.trainable = False
l.name = 'global_'+l.name
for l in local_branch_model.layers:
l.name = 'local_'+l.name
l.trainable = False

global_pooling = global_branch_model.get_layer('global_global_average_pooling2d_1')

local_pooling = local_branch_model.get_layer('local_global_average_pooling2d_1')

merged = concatenate([global_pooling.output, local_pooling.output])
dense = Dense(512, activation='relu')(merged)
dropout = Dropout(self.hyperparameters.dropout)(dense)
out = Dense(1, activation='sigmoid')(dropout)

fusion_model = Model(inputs=[global_branch_model.input, local_branch_model.input], outputs=out)
loss_function = unweighted_binary_crossentropy

fusion_model.compile(optimizer=optimizer, loss=loss_function)

fusion_model.fit_generator(
generator=FusionDataGenSequence(self.labels, self.partition['train'],
current_state='train',
batch_size=self.batch_size,
hyperparameters=self.hyperparameters,
num_classes=self.number_of_classes),
epochs=self.hyperparameters.epochs,
verbose=1,
callbacks=callbacks,
workers=self.num_workers,
# max_queue_size=32,
# shuffle=False,
validation_data=FusionDataGenSequence(self.labels,
self.partition['valid'],
current_state='validation',
batch_size=self.batch_size,
hyperparameters=self.hyperparameters,
num_classes=self.number_of_classes)
# validation_steps=1
)


• The connection is here: global_pooling = global_branch_model.get_layer('global_global_average_pooling2d_1') local_pooling = local_branch_model.get_layer('local_global_average_pooling2d_1') After I generated the cropped dataset, I used that as input to the local branch. I did not measure my accuracy, only the val_loss and F1. I tried 5e-5 and 1e-7. – Tenescu Andrei Dec 24 '18 at 8:09