I'm trying to understand the hidden layers of neural networks. Input layer section covers the steps that I use before passing information to hidden layer where concerns appear.
Input Layer:
From my understanding the first step in the neural network is to weight inputs by picking the "best" linear function.
Methods used for finding optimal weights (may not be relevant to problem):
For finding optimal weights solving quadratic minimization problem is what I usually do, perhaps by finding global minimum of the convex (quadratic error $||Ax-b||^2$ in this case, $A$ being basis matrix and $b$ being ideal vector) function (since every critical point is global minimum there) or orthogonal projection equations (since error is orthogonal to the column space of basis, and their dot product gives us the equation - $A^T(Ax-b$), equating it to $x$: $x = (A^TA)^{-1}A^Tb$.
Once best linear coefficients are found (say $m$ as slope and $c$ as bias), next step is to weight the inputs by evaluating them into the function $f(x)=mx+c$.
Hidden Layer:
After all inputs are weighted, they must be summed up which gives us a constant number.
I understand that this constant number must then be inputted to some activation function (perhaps sigmoid: $f(x)=\frac{1}{(1+e^{-x})}$ or reLU: $f(x) = max(0, x)$.
Representation of simple neural network with single hidden layer having sigmoid as activation function:
Problem:
But constants that are evaluated in these activation functions obviously output constants again, so how can they be utilized for data prediction?
For example, say sum of all weighted inputs is $15$, then its evaluation in sigmoid function will be $\frac{1}{(1-e^{-15})} = 0.99999969409$. That constant number can't be utilized for data prediction (classification/regression) so what are next steps to take? If activation function returns a constant number, how can the data be predicted for different input variables?
Am I having incorrect perspective for activation functions?
