Activation Function

I am very new to machine learning and made an experiment myself. I have a few questions:

• Can I use $$Y = sin(x)$$ or $$Y = 2x$$ as an activation function for a neural network?
• Is it necessary to increase the size of a convolutional kernel to improve the effect of a convolutional neural network (CNN)?

The main idea behind the activation is that they are non-linear to break the linearity of the network. So they always have the following properties:

• Non-linear
• Injective
• Differentiable thus Continuous (not sure if this is mathematically correct)

Just a quick reminder of how a neuron is constructed: So the output is basically $$y = ActivationF(\sum_{i=1}^{m}x_i\omega_i + b)$$. Where $$\omega_n$$ and $$b$$ are parameters of the neural network.

If you use $$y = 2x$$ as an activation function, it does not break the linearity and thus is useless because it is equivalent to multiplying all the weights $$\omega_n$$ and $$b$$ by 2, and the network is going to adjust these parameters anyway.

If you use $$y = sin(x)$$, then you break the linearity, which is a good thing, but the problem is now 2 different inputs may have the same output, which is kind of weird and will make learning chaotic for your network.

Do not forget that your network will have to adjust the weights to get the right output, so it is much easier for it to adjust when it has a very smooth function like tanh rather than with a function that has waves with a random periodicity. Now I will tell you how most data scientists actually use activation functions (myself included): we just brain-deadly use the ReLU function everywhere as the activation because it gives good results everywhere and we can focus on other parts of the architecture rather than activation functions.

It is absolutly not necessary to increase the kernel size to get good results and most networks used in Computer Vision (CV) like VGG or ResNet use 3x3 kernels in most layers (sometimes 5x5 or 7x7 in the first layers).

According to my understanding of CV networks, the most important part in CNNs is that the resolution of your image after your convolutions is super low (input image may be 256x256, it should be something like 8x8 or 4x4 on the lower resolution layers of your network so the 3x3 convolution get a grasp of the whole image.

This is just my opinion and I have nothing to back it up except intuition, so do not take these last lines for granted.

Hope it is clearer now, if you have any question feel free to ask.

• I wonder how useful sine would be if we expect periodic behavior, particularly if we want to extrapolate.
– Dave
Aug 13 '21 at 16:18
• @Dave It might be worth giving it a shot, but I wound't expect it to work as if your network gets to the top of the curve (sin(pi/2)), the derivative should be 0, so your network won't be able to update the weights again and your neuron won't be able to learn anything, turning into a dead neuron. Aug 13 '21 at 21:41
• That seems more like a limitation of the optimization procedure, though, not of the mathematics of sine as an activation function. Certainly a practitioner needs to consider the optimization procedure, but the mathematician and statistician in me dislikes this fact. // I missed this last month, but ReLU isn't differentiable everywhere, but you are correct that differentiability implies continuity.
– Dave
Sep 23 '21 at 20:52