How to add the Luong Attention Mechanism into CNN?

Question

As I write my CNN model for an image binary classification below, I'm trying to add an attention layer to this model. I read from tf.keras.layers.Attention: https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention

But I still don't know exactly how to use it, any help is appreciated.

model = keras.Sequential()

model.add(Conv2D(filters = 64, kernel_size = (3, 3), activation = 'relu', padding='same', input_shape = ((256,256,3))))
model.add(MaxPooling2D(pool_size = (2, 2), strides=(2, 2)))

model.add(Conv2D(filters = 128, kernel_size = (3, 3), activation = 'relu', padding='same'))
model.add(MaxPooling2D(pool_size = (2, 2), strides=(2, 2)))

model.add(Conv2D(filters = 256, kernel_size = (3, 3), activation = 'relu', padding='same'))
model.add(MaxPooling2D(pool_size = (2, 2), strides=(2, 2)))

model.add(Flatten())

model.add(Dense(units = 512, activation = 'relu'))
model.add(Dropout(rate = 0.5))

model.add(Dense(units = 1,activation='sigmoid'))

Answer 1

TL;DR: I find it easier to use through Keras's Functional API

One example, given your code, can be:

inputs = Input((256, 256, 3))

conv1 = Conv2D(filters = 64, kernel_size = (3, 3), activation = 'relu', padding='same')(inputs)
maxpool1 = MaxPooling2D(pool_size = (2, 2), strides=(2, 2))(conv1)

# attn1 = Attention(maxpool1, ))

conv2 = Conv2D(filters = 128, kernel_size = (3, 3), activation = 'relu', padding='same')(maxpool1)
maxpool2 = MaxPooling2D(pool_size = (2, 2), strides=(2, 2))(conv2)

conv3 = Conv2D(filters = 256, kernel_size = (3, 3), activation = 'relu', padding='same')(maxpool2)
maxpool3 = MaxPooling2D(pool_size = (2, 2), strides=(2, 2))(conv3)

attn = Attention()([maxpool3, maxpool3])

flattened = Flatten()(attn)

fc1 = Dense(units = 512, activation = 'relu')(flattened)
drop1 = Dropout(rate = 0.5)(fc1)

final = Dense(units = 1,activation='sigmoid')(drop1)

model = Model(inputs=inputs, outputs=final)

Long story:

Normally, Luong attention needs two layers/their weights to compute a mapping between the two, an "encoder" and a "decoder". One could say that any two sequential layers can be treated as an encoder and a decoder. For your computer vision model, we can do something like "self-attention" with the 3rd Conv block.

That's why I did Attention()([maxpool3, maxpool3]) in the code above.

You can also try doing this with a dense layer, like Transformers, maybe it will work.

Just in case, check this link to learn more about attention. Personally, I find it useful in deciding how to use attention.

Now, using attention like that for CV models might not be the most optimal approach. There are other similar ideas, but specific for computer vision models. The most popular such idea is Squeeze and Excitation networks. It's easy to implement and works well. Another, more recent idea is the Convolutional Block Attention Module (CBAM), which contains two types of attention, and is also pretty easy to implement.