# Properly using activation functions of neural network

I'm trying to understand the hidden layers of neural networks. Input layer section covers the steps that I use before passing information to hidden layer where concerns appear.

# Input Layer:

From my understanding the first step in the neural network is to weight inputs by picking the "best" linear function.

Methods used for finding optimal weights (may not be relevant to problem):

For finding optimal weights solving quadratic minimization problem is what I usually do, perhaps by finding global minimum of the convex (quadratic error $||Ax-b||^2$ in this case, $A$ being basis matrix and $b$ being ideal vector) function (since every critical point is global minimum there) or orthogonal projection equations (since error is orthogonal to the column space of basis, and their dot product gives us the equation - $A^T(Ax-b$), equating it to $x$: $x = (A^TA)^{-1}A^Tb$.

Once best linear coefficients are found (say $m$ as slope and $c$ as bias), next step is to weight the inputs by evaluating them into the function $f(x)=mx+c$.

# Hidden Layer:

After all inputs are weighted, they must be summed up which gives us a constant number.

I understand that this constant number must then be inputted to some activation function (perhaps sigmoid: $f(x)=\frac{1}{(1+e^{-x})}$ or reLU: $f(x) = max(0, x)$.

Representation of simple neural network with single hidden layer having sigmoid as activation function:

Picture reference

# Problem:

But constants that are evaluated in these activation functions obviously output constants again, so how can they be utilized for data prediction?

For example, say sum of all weighted inputs is $15$, then its evaluation in sigmoid function will be $\frac{1}{(1-e^{-15})} = 0.99999969409$. That constant number can't be utilized for data prediction (classification/regression) so what are next steps to take? If activation function returns a constant number, how can the data be predicted for different input variables?

Am I having incorrect perspective for activation functions?

• Have you seen here? The purpose of activation functions is to add nonlinearity to estimate non-linear functions. If you don't use non-linear activation function, you just estimate simple hyper-planes. Jul 7, 2018 at 18:42
• Also, take a look at Tensorflow Playground. It has shown the weights, inputs, and outputs. Jul 7, 2018 at 19:11
• @Media Yes, I knew the purpose of activation function was to "squash" the linearity. Thanks for your answer, I think now I fully understand the basic concept of activation functions, that it's output must be classified in binary. In linear regression, data can be predicted by the optimal linear function. How is data predicted from activation functions? (considering that it returns constant on weighted input sum). If the weighted inputs sum is the same, what's the point of multiple hidden layers? Shouldn't I find global minimum of error function for weights? Thanks! Jul 7, 2018 at 19:26

How is data predicted from activation functions? (considering that it returns constants on weighted input sum).

You should consider the fact that the label of your input data is going to be predicted by the network. Moreover, the outputs of the network usually represent the probability of belonging to each class. The label is not predicted by the activation function, it is predicted by the network. Take a network as a mapper which can map the inputs to outputs. The activations are used to add non-linearity in order to approximate complicated non-linear mappings.

What's the point of multiple hidden layers?

As you can read here, the purpose of adding those is that you can learn different complicated regions. Another interpretation can be this that by adding more layers you can learn more complicated mappings which have non-linear behaviour.

About the global minimum, because you can not see the cost function which is the error based on the weights, due to the fact that it is not possible to visualise it because it has so many dimensions, you should use gradient-descent based algorithms.

• Thank you for the answer. My apologies for confusion but when you mentioned neural network which part of it did you mean? Inputs are weighted and summed up and then passed to squashing function in hidden layer which eliminates non-linearity (is this process reiterated for input weight optimization?). From your tensorflow playground link I saw that activation functions are weighted too. Jul 8, 2018 at 10:01
• About your first part of the comment, neural nets can have one neuron or multiple neurons within different layers. Are you familiar with MLP? Jul 8, 2018 at 10:12
• No, unfortunately I'm not. I'm trying to understand how single neuron separator works before switching to multilayer perceptron. I thought inputs were weighted by optimal linear function (if data is small by normal equations), then summed up and passed to squashing function which has output that is classified between -1, 0 and 1, is this correct? I'm guessing multi-layer perceptrons are way to empty the confusion. Thanks a lot. Jul 8, 2018 at 10:27
• @ShellRox I can understand due to the fact that I've had those difficulties too :) Welcome to our community. I first recommend you taking a look at here, here and here. Jul 8, 2018 at 10:32
• I understood linear regression in-depth from Professor Gilbert Strang's linear algebra course but neural networks obviously seem to have different concept. Thanks for the links and course they will be very helpful! Jul 8, 2018 at 10:49