1
$\begingroup$

I am very new to machine learning and made an experiment myself. I have a few questions:

  • Can I use $Y = sin(x)$ or $Y = 2x$ as an activation function for a neural network?
  • Is it necessary to increase the size of a convolutional kernel to improve the effect of a convolutional neural network (CNN)?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

2 questions, 2 answers:

About the activation functions

The main idea behind the activation is that they are non-linear to break the linearity of the network. So they always have the following properties:

  • Non-linear
  • Injective
  • Differentiable thus Continuous (not sure if this is mathematically correct)

Just a quick reminder of how a neuron is constructed:

Neuron

So the output is basically $y = ActivationF(\sum_{i=1}^{m}x_i\omega_i + b)$. Where $\omega_n$ and $b$ are parameters of the neural network.

If you use $y = 2x$ as an activation function, it does not break the linearity and thus is useless because it is equivalent to multiplying all the weights $\omega_n$ and $b$ by 2, and the network is going to adjust these parameters anyway.

If you use $y = sin(x)$, then you break the linearity, which is a good thing, but the problem is now 2 different inputs may have the same output, which is kind of weird and will make learning chaotic for your network.

Do not forget that your network will have to adjust the weights to get the right output, so it is much easier for it to adjust when it has a very smooth function like tanh rather than with a function that has waves with a random periodicity.

tanh

Now I will tell you how most data scientists actually use activation functions (myself included): we just brain-deadly use the ReLU function everywhere as the activation because it gives good results everywhere and we can focus on other parts of the architecture rather than activation functions.

About convolutional kernels in CNN

It is absolutly not necessary to increase the kernel size to get good results and most networks used in Computer Vision (CV) like VGG or ResNet use 3x3 kernels in most layers (sometimes 5x5 or 7x7 in the first layers).

According to my understanding of CV networks, the most important part in CNNs is that the resolution of your image after your convolutions is super low (input image may be 256x256, it should be something like 8x8 or 4x4 on the lower resolution layers of your network so the 3x3 convolution get a grasp of the whole image.

This is just my opinion and I have nothing to back it up except intuition, so do not take these last lines for granted.

Hope it is clearer now, if you have any question feel free to ask.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I wonder how useful sine would be if we expect periodic behavior, particularly if we want to extrapolate. $\endgroup$
    – Dave
    Commented Aug 13, 2021 at 16:18
  • $\begingroup$ @Dave It might be worth giving it a shot, but I wound't expect it to work as if your network gets to the top of the curve (sin(pi/2)), the derivative should be 0, so your network won't be able to update the weights again and your neuron won't be able to learn anything, turning into a dead neuron. $\endgroup$
    – Ubikuity
    Commented Aug 13, 2021 at 21:41
  • $\begingroup$ That seems more like a limitation of the optimization procedure, though, not of the mathematics of sine as an activation function. Certainly a practitioner needs to consider the optimization procedure, but the mathematician and statistician in me dislikes this fact. // I missed this last month, but ReLU isn't differentiable everywhere, but you are correct that differentiability implies continuity. $\endgroup$
    – Dave
    Commented Sep 23, 2021 at 20:52
  • $\begingroup$ For a neural network having 2+ hidden layers and only one activation function f(x) =x, will it improve the training error if we add more layers ? $\endgroup$
    – MUK
    Commented May 16, 2022 at 12:11
  • $\begingroup$ @MUK It (almost) always improve training error to add more layers. But in your special case (your activation function is linear, it should be non linear), I doubt adding more layers would help your network learn. As always, the best thing you can do is try different configurations and check which ones gives the lowest errors. $\endgroup$
    – Ubikuity
    Commented May 17, 2022 at 17:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.