What are the filters?
A filter/kernel is a set of learnable weights which are learned using the backpropagation algorithm. You can think of each filter as storing a single template/pattern. When you convolve this filter across the corresponding input, you are basically trying to find out the similarity between the stored template and different locations in the input.
But how are they getting initialized? Do they have random initial value or there are standard image filters that are getting used?
Filters are usually initialized at a seemingly arbitrary value and then you would use a gradient descent optimizer to optimize the values so that the filters solve your problem.
There are many different initialization strategies.
- Sample from a distribution, such as a normal or uniform distribution
- Set all values to 1 or 0 or another constant
- There are also some heuristic methods that seem to work very well in practice, a popular one is the so-called glorot initializer named after Xavier Glorot who introduced them here. Glorot initializers also sample from distribution but truncate the values based on the kernel complexity.
- For specific types of kernels, there are other defaults that seem to perform well. See for example this article.
If they are getting initialized with random value then the values should get changed on the training process of the network. If that's the case then a new question is created, how does someone backpropagate the filter of the convolutional layer? What is the algorithm behind this process?
Consider the convolution operation just as a function between the input image and a matrix of random weights. As you optimize the loss function of your model, the weights (and biases) are updated such that they start forming extremely good discriminative spacial features. That is the purpose of backpropogation, which is performed with the optimizer that you defined in your model architecture. Mathematically there are a few more concepts that go into how the backprop happens on a convolution operation (full conv with 180 rotations). If you are interested then check this link.
Is the entire matrix of the output getting passed through the activation function? How does the usage of an activation function change the learning process of the convolutional layer?
Let's think of activation functions as just non-linear "scaling" functions. Given an input, the job of an activation function is to "squish" the data into a given range (example -> Relu 'squishes' the input into a range(0,inf) by simply setting every negative value to zero, and returning every positive value as is)
Now, in neural networks, activations are applied at the nodes which apply a linear function over the input feature, weight matrix, and bias (mx+c). Therefore, in the case of CNN, it's the same. Once your forward-pass takes the input image, does a convolution function over it by applying a filter (weight matrix), adds a bias, the output is then sent to an activation function to 'squish' it non-linearly before taking it to the next layer.
It's quite simple to understand why activations help. If I have a node that spits out
x1 = m0*x0+b0 and that is then sent to another node which spits out
x2 = m1*x1+b1, the overall forward pass is just
x2 = m1*(m0*x0+b0)+b1 which is the same as
x2 = (m1*m0*x0) + (m1*b0+b1) or
x2 = M*x0 + B. This shows that just stacking 2 linear equations gives another linear equation and therefore in reality there was no need for 2 nodes, instead I could have just used 1 node and used the new M and B values to get the same result x2 from x0.
This is where adding an activation function helps. Adding an activation function allows you to stack neural network layers such that you can explore the non-linear model space properly, else you would only be stuck with the
y=mx+c model space to explore because all linear combinations of linear functions is a linear model itself.
Does a convolutional layer have weight and biases like a dense layer?
Yes, it does. Its added after the weight matrix (filter) is applied to the input image using a convolution operation
Do we multiply the output matrix after the convolution process with a weight matrix and add some biases before passing it through the activation function?
A dot product operation is done between a section of the input image and the filter while convolving over the larger input image. The output matrix, is then added with bias (broadcasting) and passed through an activation function to 'squish'.
If that's true, then do we follow the same process as we do with the dense layers to train these weights and biases?
Yes, we follow the exact same process in forward pass except that there is a new operation added to the whole mix, which is convolution. It changes the dynamics especially for the backward pass but in essence, the overall intuition remains the same.
The crux for intuition is -
- Do not confuse a feature and a filter. A filter is what helps you to extract features (basic patterns) from the input image using operations such as dot, conv, bias and activations
- Each filter allows you to extract a 2D map of some simple pattern that exists over the image (such as an edge). If you have 20 filters, then you will get 20 feature maps for a 3 channel image, that are stacked as channels in the output.
- Many such features, which capture different simple patterns, are learnt as part of the training process and become the base features for the next layer (which could be another CNN or a dense)
- Combinations of these features allow you to perform your modeling task.
- The filters are trained by optimizing towards minimizing a loss function using backprop. It follows the backward reasoning:
- How can I minimize my loss?
- How can I find the best features that minimize the loss?
- How can I find the best filters that generate the best features?
- What are the best weights and biases which give me the best filters?
Here's a good reference image to keep in mind whenever working with CNNs (just to reinforce the intuition)
Hope that answers your questions.