18
$\begingroup$

I am pretty new to neural networks, but I understand linear algebra and the mathematics of convolution pretty decently.

I am trying to understand the example code I find in various places on the net for training a Keras convolutional NN with MNIST data to recognize digits. My expectation would be that when I create a convolutional layer, I would have to specify a filter or set of filters to apply to the input. But the three samples I have found all create a convolutional layer like this:

model.add(Convolution2D(nb_filter = 32, nb_row = 3, nb_col = 3,
                        border_mode='valid',
                        input_shape=input_shape))

This seems to be applying a total of 32 3x3 filters to the images processed by the CNN. But what are those filters? How would I describe them mathematically? The keras documentation is no help.

Thanks in advance,

$\endgroup$
15
$\begingroup$

By default, the filters $W$ are initialised randomly using the glorot_uniform method, which draws values from a uniform distribution with positive and negative bounds described as so: $$W \sim \mathcal{U}\left(\frac{6}{n_{in} + n_{out}}, \frac{-6}{n_{in} + n_{out}}\right),$$

where $n_{in}$ is the number of units that feed into this unit, and $n_{out}$ is the number of units this result is fed to.

When you are using the network to make a prediction, these filters are applied at each layer of the network. That is, a discrete convolution is performed for each filter on each input image, and the results of these convolutions are fed to the next layer of convolutions (or fully connected layer, or whatever else you might have).

During training, the values in the filters are optimised with backpropogation with respect to a loss function. For classification tasks such as recognising digits, usually the cross entropy loss is used. Here's a visualisation of some filters learned in the first layer (top) and the filters learned in the second layer (bottom) of a convolutional network:

conv net filters visualisation

As you can see, the first layer filters basically all act as simple edge detectors, while the second layer filters are more complex. As you go deeper into a network, the filters are able to detect more complex shapes. It gets a little tricky to visualise though, as these filters act on images that have been convolved many times already, and probably don't look much like the original natural image.

$\endgroup$
  • 5
    $\begingroup$ glorot_uniform does not use the normal distribution. I think you are describing glorot_normal. I don't think that matters greatly to the answer - the key points are random initialisation followed by effects of training. Might be worth explaining how the trained filters end up looking like edge/corner etc filters (maybe with one of the classic images of before/after training imaging first layer filters). $\endgroup$ – Neil Slater Jan 23 '17 at 14:52
  • $\begingroup$ Tim, thanks for providing the math. @Neil Slater - your insight that the filters, after training with backpropagation, might end up looking like edge detection, etc., was quite helpful. If I had more reputation, I would +1 both of your contributions. $\endgroup$ – ChrisFal Jan 23 '17 at 18:11
  • $\begingroup$ @NeilSlater Thanks for your comment -- you're right, I had confused glorot_normal and glorot_uniform, and I've updated the answer to reflect this. I also added a bit of extra info about how the filters end up, as you suggested. $\endgroup$ – timleathart Jan 23 '17 at 22:32
6
$\begingroup$

They are convolution kernels. For instance your image $A$ is 5x5, you have 32 3x3 convolution kernels $F_k$. The border_mode is 'valid' that means there is no padding around input, so the pixel (i,0),(0,j),(i,4),(4,j) are lost. Thus your results are 32 3x3 images $B_k(i,j)$, (i=1,2,3,j=1,2,3), each result image is defined by the convolution: $$ B_k(i,j) = (F_k * A) (i,j) = \sum_{l=0,1,2}\sum_{m=0,1,2}F_k(l,m)A(i-l,j-m) $$

enter image description here

The traned model will train the kernels according to your cost function, and in the end these kernels are the filters of your model.

$\endgroup$
  • $\begingroup$ I understood this math, but I'm sure many readers of this thread will find the diagram helpful. So thanks! $\endgroup$ – ChrisFal Jan 23 '17 at 18:13
  • $\begingroup$ imghost.in/images/2018/03/06/XvatD.jpg maybe picture must be with coords (0,0) on B? $\endgroup$ – vinnitu Mar 6 '18 at 8:32
  • $\begingroup$ @vinnitu yes, indeed. Actually I also need to modify the B into Bk(i,j) , (i=0,1,2,j=0,1,2). $\endgroup$ – lucky6qi Mar 7 '18 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.