8
$\begingroup$

Should there be a flat layer in between the conv layers and dense layer in YOLO?

It's something not specified in the paper, but I see most implementations of YOLO on github do this. In my implementation, I do not flatten the 7*7*1024 feature map and directly add a Dense(4096) layer after it (I'm using keras with tensorflow backend).

Code in question:

x = Conv2D(filters=1024, kernel_size=(3,3), padding='same', activation=leaky_relu)(x)
# x = Flatten()(x) # not using this right now
x = Dense(4096, activation=leaky_relu)(x)
x = Dense(30)(x)

My output of model.summary is as follows:

conv2d_294 (Conv2D)          (None, 14, 14, 1024)      9438208   
_________________________________________________________________
conv2d_295 (Conv2D)          (None, 7, 7, 1024)        9438208   
_________________________________________________________________
conv2d_296 (Conv2D)          (None, 7, 7, 1024)        9438208   
_________________________________________________________________
conv2d_297 (Conv2D)          (None, 7, 7, 1024)        9438208   
_________________________________________________________________
dense_14 (Dense)             (None, 7, 7, 4096)        4198400   
_________________________________________________________________
dense_15 (Dense)             (None, 7, 7, 30)          122910

I feel that this implementation is more correct than the flat one (unless otherwise specified by the author) because on flattening the output we lose the spatial information needed to predict bounding boxes and confidence scores for each element in the grid.

Improve this question
$\endgroup$
2
  • $\begingroup$ I feel that one of the authors of the YOLO would be best suited to answer this question of yours Joseph Redmon his mail ID [email protected] $\endgroup$
    – Santhosh
    Feb 2, 2018 at 12:15
  • $\begingroup$ @GauravKumar I am curious how you choose 4096 and 30 for the Dense layer. Would you share the reason behind? $\endgroup$
    – Cloud Cho
    Nov 19, 2021 at 19:59

3 Answers 3

Reset to default
3
$\begingroup$

Not sure that still matters for your project but it is important: the Dense layer does not flatten the entry first! It takes the last dimension of the entry tensor and connects it to the neurons of your dense layer. To be sure there is a simple thing to do: count the number of parameters of your layer. In your case, it is: 4096 (number of neurons in dense) * 1024 (last dimension of the Conv layer) + 4096 (biases of the dense neurons) = 4198400.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Yes, your statement is accurate but I think this answer is beside the point. That is not exactly what OP asked: if he/she should reduce the spatial dimensions, because using conv output with dense would only connect the channel dimension, or to flatten the conv output before dense which would not only connect the channel dimension but spatial dimensions in output of dense neurons. $\endgroup$
    – monolith
    Feb 14, 2021 at 17:33
2
$\begingroup$

It doesn't matter, with or without flattening, a Dense layer takes the whole previous layer as input. The spatial structure information is not used anymore. Some Neural Network implementations might not be able to map a spatial structure directly into a dense layer, which is why you would need the Flatten in between. Mathematically it is exactly the same in this case.

EDIT: As I expected, in the source of Keras it mentions that if the rank is higher like with convolutions, they implicitly Flatten it first.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Dense layer takes the whole previous layer as input Do you have any sources on this for non-flattened arrays for any of the libraries/implementations? Thanks. $\endgroup$
    – Gaurav Kumar
    Mar 4, 2018 at 19:29
  • 1
    $\begingroup$ I assume that implicitly Keras flattens it, but I'm not sure. I'll take a look $\endgroup$
    – Jan van der Vegt
    Mar 4, 2018 at 19:51
  • 2
    $\begingroup$ "with or without flattening, a Dense layer takes the whole previous layer as input": as stated, this is incorrect. Compare keras.models.Sequential([keras.layers.Conv1D(1, 1, input_shape=(3, 4)), keras.layers.Flatten(), keras.layers.Dense(5)]).summary() vs keras.models.Sequential([keras.layers.Conv1D(1, 1, input_shape=(3, 4)), keras.layers.Dense(5)]).summary() - the two models have different numbers of parameters. $\endgroup$
    – bers
    Nov 18, 2019 at 10:26
  • 1
    $\begingroup$ It sure looks like Dense() is doing a Conv2D with a 1x1 kernel ie "fully connecting" the last dimension (channel) of input and output tensors. $\endgroup$
    – Robert Lugg
    Mar 10, 2020 at 23:35
  • 1
    $\begingroup$ In the current version of Keras the input is not flattened, instead the Dense layer is applied repeatedly over the last dimension of the input. So for an input shape (None, 14, 14, 1024), a Dense(32) layer will apply the same matrix product over the 1024 channels inside each of the 14x14 input pixels, producing a shape (None, 14, 14, 32). github.com/keras-team/keras/blob/master/keras/layers/… $\endgroup$
    – albarji
    Feb 6, 2021 at 10:51
1
$\begingroup$

At coursera, the homework of third week of convolutional networks by professor Andrew Ng, is about this. I recommend you to see that homework. It also implements the YOLO paper. I can't add the code here, but the architecture of the network is as follows:

____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
input_1 (InputLayer)             (None, 608, 608, 3)   0                                            
____________________________________________________________________________________________________
conv2d_1 (Conv2D)                (None, 608, 608, 32)  864         input_1[0][0]                    
____________________________________________________________________________________________________
batch_normalization_1 (BatchNorm (None, 608, 608, 32)  128         conv2d_1[0][0]                   
____________________________________________________________________________________________________
leaky_re_lu_1 (LeakyReLU)        (None, 608, 608, 32)  0           batch_normalization_1[0][0]      
____________________________________________________________________________________________________
max_pooling2d_1 (MaxPooling2D)   (None, 304, 304, 32)  0           leaky_re_lu_1[0][0]              
____________________________________________________________________________________________________
conv2d_2 (Conv2D)                (None, 304, 304, 64)  18432       max_pooling2d_1[0][0]            
____________________________________________________________________________________________________
batch_normalization_2 (BatchNorm (None, 304, 304, 64)  256         conv2d_2[0][0]                   
____________________________________________________________________________________________________
leaky_re_lu_2 (LeakyReLU)        (None, 304, 304, 64)  0           batch_normalization_2[0][0]      
____________________________________________________________________________________________________
max_pooling2d_2 (MaxPooling2D)   (None, 152, 152, 64)  0           leaky_re_lu_2[0][0]              
____________________________________________________________________________________________________
conv2d_3 (Conv2D)                (None, 152, 152, 128) 73728       max_pooling2d_2[0][0]            
____________________________________________________________________________________________________
batch_normalization_3 (BatchNorm (None, 152, 152, 128) 512         conv2d_3[0][0]                   
____________________________________________________________________________________________________
leaky_re_lu_3 (LeakyReLU)        (None, 152, 152, 128) 0           batch_normalization_3[0][0]      
____________________________________________________________________________________________________
conv2d_4 (Conv2D)                (None, 152, 152, 64)  8192        leaky_re_lu_3[0][0]              
____________________________________________________________________________________________________
batch_normalization_4 (BatchNorm (None, 152, 152, 64)  256         conv2d_4[0][0]                   
____________________________________________________________________________________________________
leaky_re_lu_4 (LeakyReLU)        (None, 152, 152, 64)  0           batch_normalization_4[0][0]      
____________________________________________________________________________________________________
conv2d_5 (Conv2D)                (None, 152, 152, 128) 73728       leaky_re_lu_4[0][0]              
____________________________________________________________________________________________________
batch_normalization_5 (BatchNorm (None, 152, 152, 128) 512         conv2d_5[0][0]                   
____________________________________________________________________________________________________
leaky_re_lu_5 (LeakyReLU)        (None, 152, 152, 128) 0           batch_normalization_5[0][0]      
____________________________________________________________________________________________________
max_pooling2d_3 (MaxPooling2D)   (None, 76, 76, 128)   0           leaky_re_lu_5[0][0]              
____________________________________________________________________________________________________
conv2d_6 (Conv2D)                (None, 76, 76, 256)   294912      max_pooling2d_3[0][0]            
____________________________________________________________________________________________________
batch_normalization_6 (BatchNorm (None, 76, 76, 256)   1024        conv2d_6[0][0]                   
____________________________________________________________________________________________________
leaky_re_lu_6 (LeakyReLU)        (None, 76, 76, 256)   0           batch_normalization_6[0][0]      
____________________________________________________________________________________________________
conv2d_7 (Conv2D)                (None, 76, 76, 128)   32768       leaky_re_lu_6[0][0]              
____________________________________________________________________________________________________
batch_normalization_7 (BatchNorm (None, 76, 76, 128)   512         conv2d_7[0][0]                   
____________________________________________________________________________________________________
leaky_re_lu_7 (LeakyReLU)        (None, 76, 76, 128)   0           batch_normalization_7[0][0]      
____________________________________________________________________________________________________
conv2d_8 (Conv2D)                (None, 76, 76, 256)   294912      leaky_re_lu_7[0][0]              
____________________________________________________________________________________________________
batch_normalization_8 (BatchNorm (None, 76, 76, 256)   1024        conv2d_8[0][0]                   
____________________________________________________________________________________________________
leaky_re_lu_8 (LeakyReLU)        (None, 76, 76, 256)   0           batch_normalization_8[0][0]      
____________________________________________________________________________________________________
max_pooling2d_4 (MaxPooling2D)   (None, 38, 38, 256)   0           leaky_re_lu_8[0][0]              
____________________________________________________________________________________________________
conv2d_9 (Conv2D)                (None, 38, 38, 512)   1179648     max_pooling2d_4[0][0]            
____________________________________________________________________________________________________
batch_normalization_9 (BatchNorm (None, 38, 38, 512)   2048        conv2d_9[0][0]                   
____________________________________________________________________________________________________
leaky_re_lu_9 (LeakyReLU)        (None, 38, 38, 512)   0           batch_normalization_9[0][0]      
____________________________________________________________________________________________________
conv2d_10 (Conv2D)               (None, 38, 38, 256)   131072      leaky_re_lu_9[0][0]              
____________________________________________________________________________________________________
batch_normalization_10 (BatchNor (None, 38, 38, 256)   1024        conv2d_10[0][0]                  
____________________________________________________________________________________________________
leaky_re_lu_10 (LeakyReLU)       (None, 38, 38, 256)   0           batch_normalization_10[0][0]     
____________________________________________________________________________________________________
conv2d_11 (Conv2D)               (None, 38, 38, 512)   1179648     leaky_re_lu_10[0][0]             
____________________________________________________________________________________________________
batch_normalization_11 (BatchNor (None, 38, 38, 512)   2048        conv2d_11[0][0]                  
____________________________________________________________________________________________________
leaky_re_lu_11 (LeakyReLU)       (None, 38, 38, 512)   0           batch_normalization_11[0][0]     
____________________________________________________________________________________________________
conv2d_12 (Conv2D)               (None, 38, 38, 256)   131072      leaky_re_lu_11[0][0]             
____________________________________________________________________________________________________
batch_normalization_12 (BatchNor (None, 38, 38, 256)   1024        conv2d_12[0][0]                  
____________________________________________________________________________________________________
leaky_re_lu_12 (LeakyReLU)       (None, 38, 38, 256)   0           batch_normalization_12[0][0]     
____________________________________________________________________________________________________
conv2d_13 (Conv2D)               (None, 38, 38, 512)   1179648     leaky_re_lu_12[0][0]             
____________________________________________________________________________________________________
batch_normalization_13 (BatchNor (None, 38, 38, 512)   2048        conv2d_13[0][0]                  
____________________________________________________________________________________________________
leaky_re_lu_13 (LeakyReLU)       (None, 38, 38, 512)   0           batch_normalization_13[0][0]     
____________________________________________________________________________________________________
max_pooling2d_5 (MaxPooling2D)   (None, 19, 19, 512)   0           leaky_re_lu_13[0][0]             
____________________________________________________________________________________________________
conv2d_14 (Conv2D)               (None, 19, 19, 1024)  4718592     max_pooling2d_5[0][0]            
____________________________________________________________________________________________________
batch_normalization_14 (BatchNor (None, 19, 19, 1024)  4096        conv2d_14[0][0]                  
____________________________________________________________________________________________________
leaky_re_lu_14 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_14[0][0]     
____________________________________________________________________________________________________
conv2d_15 (Conv2D)               (None, 19, 19, 512)   524288      leaky_re_lu_14[0][0]             
____________________________________________________________________________________________________
batch_normalization_15 (BatchNor (None, 19, 19, 512)   2048        conv2d_15[0][0]                  
____________________________________________________________________________________________________
leaky_re_lu_15 (LeakyReLU)       (None, 19, 19, 512)   0           batch_normalization_15[0][0]     
____________________________________________________________________________________________________
conv2d_16 (Conv2D)               (None, 19, 19, 1024)  4718592     leaky_re_lu_15[0][0]             
____________________________________________________________________________________________________
batch_normalization_16 (BatchNor (None, 19, 19, 1024)  4096        conv2d_16[0][0]                  
____________________________________________________________________________________________________
leaky_re_lu_16 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_16[0][0]     
____________________________________________________________________________________________________
conv2d_17 (Conv2D)               (None, 19, 19, 512)   524288      leaky_re_lu_16[0][0]             
____________________________________________________________________________________________________
batch_normalization_17 (BatchNor (None, 19, 19, 512)   2048        conv2d_17[0][0]                  
____________________________________________________________________________________________________
leaky_re_lu_17 (LeakyReLU)       (None, 19, 19, 512)   0           batch_normalization_17[0][0]     
____________________________________________________________________________________________________
conv2d_18 (Conv2D)               (None, 19, 19, 1024)  4718592     leaky_re_lu_17[0][0]             
____________________________________________________________________________________________________
batch_normalization_18 (BatchNor (None, 19, 19, 1024)  4096        conv2d_18[0][0]                  
____________________________________________________________________________________________________
leaky_re_lu_18 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_18[0][0]     
____________________________________________________________________________________________________
conv2d_19 (Conv2D)               (None, 19, 19, 1024)  9437184     leaky_re_lu_18[0][0]             
____________________________________________________________________________________________________
batch_normalization_19 (BatchNor (None, 19, 19, 1024)  4096        conv2d_19[0][0]                  
____________________________________________________________________________________________________
conv2d_21 (Conv2D)               (None, 38, 38, 64)    32768       leaky_re_lu_13[0][0]             
____________________________________________________________________________________________________
leaky_re_lu_19 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_19[0][0]     
____________________________________________________________________________________________________
batch_normalization_21 (BatchNor (None, 38, 38, 64)    256         conv2d_21[0][0]                  
____________________________________________________________________________________________________
conv2d_20 (Conv2D)               (None, 19, 19, 1024)  9437184     leaky_re_lu_19[0][0]             
____________________________________________________________________________________________________
leaky_re_lu_21 (LeakyReLU)       (None, 38, 38, 64)    0           batch_normalization_21[0][0]     
____________________________________________________________________________________________________
batch_normalization_20 (BatchNor (None, 19, 19, 1024)  4096        conv2d_20[0][0]                  
____________________________________________________________________________________________________
space_to_depth_x2 (Lambda)       (None, 19, 19, 256)   0           leaky_re_lu_21[0][0]             
____________________________________________________________________________________________________
leaky_re_lu_20 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_20[0][0]     
____________________________________________________________________________________________________
concatenate_1 (Concatenate)      (None, 19, 19, 1280)  0           space_to_depth_x2[0][0]          
                                                                   leaky_re_lu_20[0][0]             
____________________________________________________________________________________________________
conv2d_22 (Conv2D)               (None, 19, 19, 1024)  11796480    concatenate_1[0][0]              
____________________________________________________________________________________________________
batch_normalization_22 (BatchNor (None, 19, 19, 1024)  4096        conv2d_22[0][0]                  
____________________________________________________________________________________________________
leaky_re_lu_22 (LeakyReLU)       (None, 19, 19, 1024)  0           batch_normalization_22[0][0]     
____________________________________________________________________________________________________
conv2d_23 (Conv2D)               (None, 19, 19, 425)   435625      leaky_re_lu_22[0][0]             
====================================================================================================
Total params: 50,983,561
Trainable params: 50,962,889
Non-trainable params: 20,672

Like you, I can't see any flatten layer.

Improve this answer
$\endgroup$
4
  • $\begingroup$ what is the concatenate layer and which Yolo version is this? Seems like YoloV3 to me. Also why use the batch normalization before activation functions? $\endgroup$
    – monolith
    Feb 14, 2021 at 17:08
  • 1
    $\begingroup$ @monolith About the version, I'm not sure. This is not a problem I'm currently working on. This also holds for concat layer. About batch normalization, there are two approaches. Some believe applying that before activation is better than applying that after activation function. Pr. Ng has elaborated that in his deep learning course. You can also see this trend in ResNet. I hope this helps you. If you insist knowing what is that concat, I beileve there are many documented open source implementations. $\endgroup$
    – Green Falcon
    Feb 14, 2021 at 17:53
  • $\begingroup$ I think I know what the concat layer is: They added this in yolo v3 to get the feature maps from lower layers thus making it easier to learn small object representations, because they get lost as spatial resolution decreases. I was just confused because it's YoloV1 question. Nvm, thanks. $\endgroup$
    – monolith
    Feb 14, 2021 at 18:49
  • $\begingroup$ @monolith Thank you! $\endgroup$
    – Green Falcon
    Feb 15, 2021 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.