Skip to main content
Data Science

Return to Answer

Corrected last typo, thanks Ethan for the edit !
Source Link
Ubikuity
Ubikuity
  • 636
  • 4
  • 9

2 questions, 2 answers:

About the activation functions

The main idea behind the activation is that they are non-linear to break the linearity of the network. So they always have the following properties:

  • Non-linear
  • Injective
  • Differentiable thus Continuous (not sure if this is mathematically correct)

Just a quick reminder of how a neuron is constructed:

Neuron

So the output is basically $y = ActivationF(\sum_{i=1}^{m}x_i\omega_i + b)$. Where $\omega_n$ and $b$ are parameters of the neural network.

If you use $y = 2x$ as an activation function, it does not break the linearity and thus is useless because it is equivalent to multiplying all the weights $\omega_n$ and $b$ by 2, and the network is going to adjust these parameters anyway.

If you use $y = sin(x)$, then you break the linearity, which is a good thing, but the problem is now 2 different inputs may have the same output, which is kind of weird and will make learning chaotic for your network.

Do not forget that your network will have to adjust the weights to get the right output, so it is much easier for it to adjust when isit has a very smooth function like tanh rather than with a function that has waves with a random periodicity.

tanh

Now I will tell you how most data scientists actually use activation functions (myself included): we just brain-deadly use the ReLU function everywhere as the activation because it gives good results everywhere and we can focus on other parts of the architecture rather than activation functions.

About convolutional kernels in CNN

It is absolutly not necessary to increase the kernel size to get good results and most networks used in Computer Vision (CV) like VGG or ResNet use 3x3 kernels in most layers (sometimes 5x5 or 7x7 in the first layers).

According to my understanding of CV networks, the most important part in CNNs is that the resolution of your image after your convolutions is super low (input image may be 256x256, it should be something like 8x8 or 4x4 on the lower resolution layers of your network so the 3x3 convolution get a grasp of the whole image.

This is just my opinion and I have nothing to back it up except intuition, so do not take these last lines for granted.

Hope it is clearer now, if you have any question feel free to ask.

2 questions, 2 answers:

About the activation functions

The main idea behind the activation is that they are non-linear to break the linearity of the network. So they always have the following properties:

  • Non-linear
  • Injective
  • Differentiable thus Continuous (not sure if this is mathematically correct)

Just a quick reminder of how a neuron is constructed:

Neuron

So the output is basically $y = ActivationF(\sum_{i=1}^{m}x_i\omega_i + b)$. Where $\omega_n$ and $b$ are parameters of the neural network.

If you use $y = 2x$ as an activation function, it does not break the linearity and thus is useless because it is equivalent to multiplying all the weights $\omega_n$ and $b$ by 2, and the network is going to adjust these parameters anyway.

If you use $y = sin(x)$, then you break the linearity, which is a good thing, but the problem is now 2 different inputs may have the same output, which is kind of weird and will make learning chaotic for your network.

Do not forget that your network will have to adjust the weights to get the right output, so it is much easier for it to adjust when is has a very smooth function like tanh rather than with a function that has waves with a random periodicity.

tanh

Now I will tell you how most data scientists actually use activation functions (myself included): we just brain-deadly use the ReLU function everywhere as the activation because it gives good results everywhere and we can focus on other parts of the architecture rather than activation functions.

About convolutional kernels in CNN

It is absolutly not necessary to increase the kernel size to get good results and most networks used in Computer Vision (CV) like VGG or ResNet use 3x3 kernels in most layers (sometimes 5x5 or 7x7 in the first layers).

According to my understanding of CV networks, the most important part in CNNs is that the resolution of your image after your convolutions is super low (input image may be 256x256, it should be something like 8x8 or 4x4 on the lower resolution layers of your network so the 3x3 convolution get a grasp of the whole image.

This is just my opinion and I have nothing to back it up except intuition, so do not take these last lines for granted.

Hope it is clearer now, if you have any question feel free to ask.

2 questions, 2 answers:

About the activation functions

The main idea behind the activation is that they are non-linear to break the linearity of the network. So they always have the following properties:

  • Non-linear
  • Injective
  • Differentiable thus Continuous (not sure if this is mathematically correct)

Just a quick reminder of how a neuron is constructed:

Neuron

So the output is basically $y = ActivationF(\sum_{i=1}^{m}x_i\omega_i + b)$. Where $\omega_n$ and $b$ are parameters of the neural network.

If you use $y = 2x$ as an activation function, it does not break the linearity and thus is useless because it is equivalent to multiplying all the weights $\omega_n$ and $b$ by 2, and the network is going to adjust these parameters anyway.

If you use $y = sin(x)$, then you break the linearity, which is a good thing, but the problem is now 2 different inputs may have the same output, which is kind of weird and will make learning chaotic for your network.

Do not forget that your network will have to adjust the weights to get the right output, so it is much easier for it to adjust when it has a very smooth function like tanh rather than with a function that has waves with a random periodicity.

tanh

Now I will tell you how most data scientists actually use activation functions (myself included): we just brain-deadly use the ReLU function everywhere as the activation because it gives good results everywhere and we can focus on other parts of the architecture rather than activation functions.

About convolutional kernels in CNN

It is absolutly not necessary to increase the kernel size to get good results and most networks used in Computer Vision (CV) like VGG or ResNet use 3x3 kernels in most layers (sometimes 5x5 or 7x7 in the first layers).

According to my understanding of CV networks, the most important part in CNNs is that the resolution of your image after your convolutions is super low (input image may be 256x256, it should be something like 8x8 or 4x4 on the lower resolution layers of your network so the 3x3 convolution get a grasp of the whole image.

This is just my opinion and I have nothing to back it up except intuition, so do not take these last lines for granted.

Hope it is clearer now, if you have any question feel free to ask.

deleted 9 characters in body
Source Link
Ethan
Ethan
  • 1.7k
  • 9
  • 24
  • 39

2 questions, 2 answers  :

About the activation functions

The main idea behind the activation is that they are non-linear to break the linearity of the network. So they always have the following properties  :

  • Non-linear
  • Injective
  • Differentiable thus Continuous (not sure if this is mathematically correct)

justJust a quick reminder of how a neuron is constructed  :

Neuron

So the output is basically $y = ActivationF(\sum_{i=1}^{m}x_i\omega_i + b)$. Where $\omega_n$ and $b$ are parameters of the neural network.

If you use y = 2x$y = 2x$ as an activation function, it does not break the linearity and thus is useless because it is equivalent asto multiplying all the weights $\omega_n$ and $b$ by 2, and the network is going to adjust these parameters anyway.

If you use $y = sin(x)$, then you break the linearity, which is a good thing, but the problem is now 2 different inputinputs may have the same output, which is kindakind of weird and will make learning chaotic for your network.

Don'tDo not forget that your network is will have to adjust the weights to get the right output, so it is much easier for it to adjust when having a function that is has a very smooth function like tanh rather than with a function that doeshas waves with a random periodicity.

tanh

Now I'llI will tell you how most Datascientistdata scientists actually use activation functions (myself included)  : we just braindeadlybrain-deadly use the ReLU function everywhere as the activation because it is the one that gives good results everywhere and we can focus on other parts of the architecture rather than activation functions.

About convolutional kernels in CNN

It is absolutly not necessary to increase the kernel size to get good results and most networks used in CVComputer Vision (Computer VisionCV) like VGG or ResNet use 3x3 kernels in most layers (sometimes 5x5 or 7x7 in the first layers).

According to my understanding of CV networks, the most important part in CNNCNNs is that the resolution of your image after your convolutions is super low (input image may be 256x256, it should be something like 8x8 or 4x4 on the lower resolution layers of your network so the 3x3 convolution get a grasp of the whole image.

This is just my opinion and I have nothing to back it up except intuition, so do not take these last lines for granted.

Hope it is clearer now, if you have any question feel free to ask.

2 questions, 2 answers  :

About the activation functions

The main idea behind the activation is that they are non-linear to break the linearity of the network. So they always have the following properties  :

  • Non-linear
  • Injective
  • Differentiable thus Continuous (not sure if this is mathematically correct)

just a quick reminder of how a neuron is constructed  :

Neuron

So the output is basically $y = ActivationF(\sum_{i=1}^{m}x_i\omega_i + b)$. Where $\omega_n$ and $b$ are parameters of the neural network.

If you use y = 2x as activation function, it does not break the linearity and thus is useless because it is equivalent as multiplying all the weights $\omega_n$ and $b$ by 2, and the network is going to adjust these parameters anyway.

If you use $y = sin(x)$, then you break the linearity, which is a good thing, but the problem is now 2 different input may have the same output, which is kinda weird and will make learning chaotic for your network.

Don't forget that your network is will have to adjust the weights to get the right output, so it is much easier for it to adjust when having a function that is very smooth like tanh rather than with a function that does waves with a random periodicity.

tanh

Now I'll tell you how most Datascientist actually use activation functions (myself included)  : we just braindeadly use ReLU function everywhere as activation because it is the one that gives good results everywhere and we can focus on other parts of the architecture rather than activation functions.

About convolutional kernels in CNN

It is absolutly not necessary to increase the kernel size to get good results and most networks used in CV (Computer Vision) like VGG or ResNet use 3x3 kernels in most layers (sometimes 5x5 or 7x7 in the first layers).

According to my understanding of CV networks, the most important part in CNN is that the resolution of your image after your convolutions is super low (input image may be 256x256, it should be something like 8x8 or 4x4 on the lower resolution layers of your network so the 3x3 convolution get a grasp of the whole image.

This is just my opinion and I have nothing to back it up except intuition, so do not take these last lines for granted.

Hope it is clearer now, if you have any question feel free to ask.

2 questions, 2 answers:

About the activation functions

The main idea behind the activation is that they are non-linear to break the linearity of the network. So they always have the following properties:

  • Non-linear
  • Injective
  • Differentiable thus Continuous (not sure if this is mathematically correct)

Just a quick reminder of how a neuron is constructed:

Neuron

So the output is basically $y = ActivationF(\sum_{i=1}^{m}x_i\omega_i + b)$. Where $\omega_n$ and $b$ are parameters of the neural network.

If you use $y = 2x$ as an activation function, it does not break the linearity and thus is useless because it is equivalent to multiplying all the weights $\omega_n$ and $b$ by 2, and the network is going to adjust these parameters anyway.

If you use $y = sin(x)$, then you break the linearity, which is a good thing, but the problem is now 2 different inputs may have the same output, which is kind of weird and will make learning chaotic for your network.

Do not forget that your network will have to adjust the weights to get the right output, so it is much easier for it to adjust when is has a very smooth function like tanh rather than with a function that has waves with a random periodicity.

tanh

Now I will tell you how most data scientists actually use activation functions (myself included): we just brain-deadly use the ReLU function everywhere as the activation because it gives good results everywhere and we can focus on other parts of the architecture rather than activation functions.

About convolutional kernels in CNN

It is absolutly not necessary to increase the kernel size to get good results and most networks used in Computer Vision (CV) like VGG or ResNet use 3x3 kernels in most layers (sometimes 5x5 or 7x7 in the first layers).

According to my understanding of CV networks, the most important part in CNNs is that the resolution of your image after your convolutions is super low (input image may be 256x256, it should be something like 8x8 or 4x4 on the lower resolution layers of your network so the 3x3 convolution get a grasp of the whole image.

This is just my opinion and I have nothing to back it up except intuition, so do not take these last lines for granted.

Hope it is clearer now, if you have any question feel free to ask.

Source Link
Ubikuity
Ubikuity
  • 636
  • 4
  • 9

2 questions, 2 answers :

About the activation functions

The main idea behind the activation is that they are non-linear to break the linearity of the network. So they always have the following properties :

  • Non-linear
  • Injective
  • Differentiable thus Continuous (not sure if this is mathematically correct)

just a quick reminder of how a neuron is constructed :

Neuron

So the output is basically $y = ActivationF(\sum_{i=1}^{m}x_i\omega_i + b)$. Where $\omega_n$ and $b$ are parameters of the neural network.

If you use y = 2x as activation function, it does not break the linearity and thus is useless because it is equivalent as multiplying all the weights $\omega_n$ and $b$ by 2, and the network is going to adjust these parameters anyway.

If you use $y = sin(x)$, then you break the linearity, which is a good thing, but the problem is now 2 different input may have the same output, which is kinda weird and will make learning chaotic for your network.

Don't forget that your network is will have to adjust the weights to get the right output, so it is much easier for it to adjust when having a function that is very smooth like tanh rather than with a function that does waves with a random periodicity.

tanh

Now I'll tell you how most Datascientist actually use activation functions (myself included) : we just braindeadly use ReLU function everywhere as activation because it is the one that gives good results everywhere and we can focus on other parts of the architecture rather than activation functions.

About convolutional kernels in CNN

It is absolutly not necessary to increase the kernel size to get good results and most networks used in CV (Computer Vision) like VGG or ResNet use 3x3 kernels in most layers (sometimes 5x5 or 7x7 in the first layers).

According to my understanding of CV networks, the most important part in CNN is that the resolution of your image after your convolutions is super low (input image may be 256x256, it should be something like 8x8 or 4x4 on the lower resolution layers of your network so the 3x3 convolution get a grasp of the whole image.

This is just my opinion and I have nothing to back it up except intuition, so do not take these last lines for granted.

Hope it is clearer now, if you have any question feel free to ask.