I heard that the activation could be thought of as a unnormalized angle between input and nodes.

How is this true?

My idea is that:

We have an input vector $\vec{x}$ per feature (?). This then gets multiplied with a dot product to a weight vector connecting the input to the neuron $\vec{w}$ and thus we have $\vec{x} \cdot \vec{w}$.

Now the dot product has to do with directions, such that it is minimized when they are opposite, and it is maximized when they are in the same direction. Thus it also has to do with the angle:

$\vec{x} \cdot \vec{w} = |\vec{x}| |\vec{w}| \cos{\theta}$

where $\theta$ si the angle between them. After the dot product we can take the $\tanh$ activation function and we would have $\tanh(\vec{x} \cdot \vec{w}) = \tanh(|\vec{x}| |\vec{w}| \cos{\theta})$

This is the further I could get. Any idea?

Also, is it correct that in the neural network we put a vector per each feature? I mean I know we should put an input neuron per each feature, so does that mean that each input neuron holds a vector? Or is the vector formulation a shortcut for the entire network?

  • 1
    $\begingroup$ The vector formulation is a shortcut for the entire layer, as each neuron has a scalar input and a scalar output. Seems to me that the angle relates to your first equation, before you take the $\tanh$. $\endgroup$ – Dave Kielpinski Oct 17 '17 at 16:47
  • $\begingroup$ Don't forget the bias, b, adding it to the dot product before applying the tanh activation function. $\endgroup$ – mjul Oct 23 '17 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.