Padding in Keras with output half sized input

Question

Here is my Keras model I'm working on:

model = Sequential()
model.add(Conv2D(64, kernel_size=(7, 7), strides = 2, padding = 3,
                 input_shape=input_shape)) # (224,224,64)

model.add(MaxPooling2D(pool_size=(2, 2), strides = 2)) # (112,112,64)
model.add(Conv2D(192, kernel_size = (3,3), padding = 1)) #(112,112,192)
model.add(MaxPooling2D(pool_size = (2,2),strides = 2)) #(56,56,192)
model.add(Conv2D(128, kernel_size = (1,1))) #(56,56,128)
model.add(Conv2D(256, kernel_size = (3,3), padding = 1)) #(56,56,256)
model.add(Conv2D(256, kernel_size = (1,1))) #(56,56,256)
model.add(Conv2D(512, kernel_size = (3,3),padding = 1)) #(56,56,512)
model.add(MaxPooling2D(pool_size = (2,2), strides = 2)) #(28,28,512)
model.add(Conv2D(256, kernel_size = (1,1))) #(28,28,128)
model.add(Conv2D(512, kernel_size = (3,3), padding = 1)) #(28,28,512)
model.add(Conv2D(256, kernel_size = (1,1))) #(28,28,128)
model.add(Conv2D(512, kernel_size = (3,3), padding = 1)) #(28,28,512)
model.add(Conv2D(256, kernel_size = (1,1))) #(28,28,128)
model.add(Conv2D(512, kernel_size = (3,3), padding = 1)) #(28,28,512)
model.add(Conv2D(256, kernel_size = (1,1))) #(28,28,128)
model.add(Conv2D(512, kernel_size = (3,3), padding = 1)) #(28,28,512)
model.add(Conv2D(512, kernel_size = (1,1))) #(28,28,512)
model.add(Conv2D(1024,kernel_size = (3,3), padding = 1)) #(28,28,1024)
model.add(MaxPooling2D(pool_size = (2,2), strides = 2)) #(14,14,1024)
model.add(Conv2D(512, kernel_size = (1,1))) #(14,14,512)
model.add(Conv2D(1024,kernel_size = (3,3), padding = 1)) #(14,14,1024)
model.add(Conv2D(512, kernel_size = (1,1))) #(14,14,512)
model.add(Conv2D(1024,kernel_size = (3,3), padding = 1)) #(14,14,1024)
model.add(Conv2D(1024, kernel_size = (3,3), padding = 1)) #(14,14,1024)
model.add(Conv2D(1024, kernel_size = (3,3), strides = 2, padding = 3)) #(7,7,1024)
model.add(Conv2D(1024,kernel_size = (3,3), padding = 1)) #(7,7,1024)
model.add(Conv2D(1024, kernel_size = (3,3), padding = 1)) #(7,7,1024)

model.add(Flatten())
model.add(Dense(4096))
model.add(Dense(7*7*30))
model.add(Reshape(7,7,30))

When I compile it, I got an error for padding because Keras knows only 'same', valid' and 'casual'. I understand these, but I really need padding somewhere to be equal to 3 because my output should be a half of input (we have strides equal to 2). I really don't know how to fix it. How to do padding if we want to half size the input with strides 2?

Answer 1

If you goal is simply to halve the size of your filters, you could think about using some different methods other than padding, such as dilated convolutions. Have a look at this paper for some ideas and nice explanations with pictures. Just thinking about your dimensions quickly, I am not sure you could go from 14 to 7 very easily. getting to 8 or 5 is simple enough though.

---

One thing that isn't always obvious if you just started learning KEras, is that you can mix in Tensorflow operations directly from the Tensorflow library in with your Keras code.

In TF there is a function called pad, which allows you to specify the padding manually on each side of a tensor. There are also options to say whether the padding is done with zeros, or if the values inside your original tensor are repeated/mirrored (using the mode argument).

You could try using this to pad the layers. I can show you how to pad the tensors to get the effect you want:

from keras.models import Sequential, Model
from keras.layers import (Input, Conv2D, MaxPooling2D, 
                          Flatten, Dense, Reshape, Lambda)
import tensorflow as tf

input_shape = (448, 448, 3)
batch_shape = (None,) + input_shape
raw_input = tf.placeholder(dtype='float16', shape=batch_shape)
paddings = tf.constant([[0, 0],   # the batch size dimension
                        [3, 3],   # top and bottom of image
                        [3, 3],   # left and right
                        [0, 0]])  # the channels dimension

padded_input = tf.pad(raw_input, paddings, mode='CONSTANT',
                      constant_values=0.0)  # pads with 0 by default

padded_shape = padded_input.shape     # (454, 454, 3) because we add 2*3 padding

input_layer = Input(padded_shape, batch_shape, tensor=padded_input)
layer0 = Conv2D(192, kernel_size = (3,3), padding='valid')(input_layer)
layer1 = MaxPooling2D(pool_size=(2, 2), strides = 2)(layer0) 
layer2 = Conv2D(192, kernel_size = (3,3), padding='valid')(layer1)
layer3 = MaxPooling2D(pool_size = (2,2),strides = 2)(layer2)
layer4 = Conv2D(512, kernel_size = (3,3), padding='valid')(layer3)
layer5 = MaxPooling2D(pool_size = (2,2), strides = 2)(layer4)
layer6 = Conv2D(256, kernel_size = (1,1), padding='valid')(layer5)
# layer6.shape --> [Dimension(None), Dimension(55), Dimension(55), Dimension(256)]

layer6_padded = tf.pad(layer6, paddings, mode='CONSTANT')
layer6_output = Input(input_shape=layer6_padded.shape)(layer6_padded)

# This will end up giving this error at compilation:
# RuntimeError: Graph disconnected: ...,

layer7 = Conv2D(1024, kernel_size=(3, 3), strides=2)(layer6_output)
layer8 = Flatten()(layer7)
layer9 = Dense(4096)(layer7)
layer10 = Dense(7*7*30)(layer8)
output_layer = Reshape((7, 7, 30))(layer10)

# The following both fail to get the graph as we would like it
model = Model(inputs=[input_layer], outputs=[output_layer])
#model = Model(inputs=[input_layer, layer6_output], outputs=[output_layer])

model.compile('adam', 'mse')
model.summary()

I have been unable to then bring this tensor back into the Keras model (as a layer, which is required) because the standard way of using the Input object forces it to be the entry point of the computational graph, but we want the padded tensors to form an intermediary layer.

If you don't force the padded tensors into a Keras layer, attributes will be missing:

# AttributeError: 'Tensor' object has no attribute '_keras_history'

Which you can hack by just adding the attribute from the layer before we padded:

#layer6_output._keras_history = layer6._keras_history

Unfortunately, I still ran into other errors.

Perhaps you can post a new question on StackOverflow asking how to to this, if you can find anything. I did have a quick try using the idea of creating two graphs and then joining them, but didn't succeed.