What does a forward pass consist of?
I understand that, during a forward pass, data is passed through the network and the network returns predicted values, but how does this work?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
If you already understand the high-level concept of Forward Pass, all that's left to understand is the Neural Net algorithm itself.
Please see the picture below, taken from here:
This is a simple representation of a neural net. On the left input is taken into the net, scaled by weight factors, and summed to give the Net Input Function. Typically, there is also a bias (constant) added to each input. Multiplying by a weight and adding a bias turns each input into a linear equation. The Net Input Function is a combination of linear equations that is used as input to the Activation Function. The Activation Function can take many different forms, but they all perform the same basic job, adding some non-linearity to the inputs. A common Activation Function is ReLU:
If you're struggling to see how this adds non-linearity, try creating and using the function in Desmos. You'll get something like the following:
The black function is how you define ReLU in Desmos. The green function is a combination of linear functions passed through ReLU.
If the Neural Net has more hidden layers, the Activation Function's output is passed forward to the next hidden layer, with a weight and bias, as before, and the process is repeated. If there are no more Hidden layers the output is summed and used to produce predicted values for the input data.
The Forward Pass's final step is to compute loss, by squaring and summing the difference between the predicted and expected values. The loss indicates the predictive power of the network on the input data.