2 questions, 2 answers:


----------


## About the activation functions ##

The main idea behind the activation is that they are non-linear to break the linearity of the network. **So they always have the following properties:**

 - Non-linear
 - Injective
 - Differentiable thus Continuous (not sure if this is mathematically correct)

Just a quick reminder of how a neuron is constructed:

[![Neuron][1]][1]

So the output is basically $y = ActivationF(\sum_{i=1}^{m}x_i\omega_i + b)$. Where $\omega_n$ and $b$ are parameters of the neural network.

If you use $y = 2x$ as an activation function, **it does not break the linearity** and thus is useless because it is equivalent to multiplying all the weights $\omega_n$ and $b$ by 2, and the network is going to adjust these parameters anyway.

If you use $y = sin(x)$, then you break the linearity, which is a good thing, but the problem is now **2 different inputs may have the same output**, which is kind of weird and will make learning chaotic for your network.

Do not forget that your network will have to adjust the weights to get the right output, so it is much easier for it to adjust when is has a very **smooth** function like tanh rather than with a function that has waves with a random periodicity.

[![tanh][2]][2]

Now I will tell you how most data scientists actually use activation functions (myself included): we just brain-deadly use the **ReLU** function everywhere as the activation because it gives good results everywhere and we can focus on other parts of the architecture rather than activation functions.

----------
## About convolutional kernels in CNN ##

It is absolutly **not necessary to increase the kernel size** to get good results and most networks used in Computer Vision (CV) like VGG or ResNet use **3x3 kernels in most layers** (sometimes 5x5 or 7x7 in the first layers).

According to my understanding of CV networks, the most important part in CNNs is that the resolution of your image after your convolutions is super low (input image may be 256x256, it should be something like 8x8 or 4x4 on the lower resolution layers of your network so the 3x3 convolution get a grasp of the whole image.

This is just my opinion and I have nothing to back it up except intuition, so do not take these last lines for granted.

Hope it is clearer now, if you have any question feel free to ask.

  [1]: https://i.sstatic.net/eXxS3.jpg
  [2]: https://i.sstatic.net/CTlks.png