I'm currently reading this book and want someone to tell me if what currently I'm assuming about neural network is right or not. https://www.pyimagesearch.com/deep-learning-computer-vision-python-book/
If a layer has 30 neurons: In a feedforward meural network each neuron of the first layer multiplied with all the neurons of the second layer. That would be 30 neurons of the first layer multiplied by the 30 neurons of the second layer that would give a total of 900 (30*30 = 900) multiplications (is this is correct?)
But those 900 multiplications is only for the first layer and the second layer of the neural network(nn). So if I have a feedforward nn that has 3 layers I would have to do 900 more multiplications because they are the multiplications of the output of the first layer (that they are the input of the second layer) with the weight of the third layer.
So to recap what I said so far I have a feedforward nn with 3 layer with 30 neurons in the first two layers multiply each neuron with all the neurons of the second layer it would be 30 * 30 = 900 multiplications for each additional layer that I add, it adds 900 additional multiplications.
Then for a model with three (3) fully connected layer would be 900 multiplications in the first two layer because for what I said earlier and 900 multiplications in the second and third layer for a total of 1,800 (900 + 900) multiplications excluding an activation function and this is only for a forward pass, is my understanding correct?
And in addition to a forward pass in a typical Neural network they also have a backward pass that because of my calculations doing earlier they are 1,800 derivatives (gradient) for the entire backward pass. Am I correct assuming this for a Neural network?. That's why a CPU computer takes so long to train a model because it has to do about 3,600 (1,800 + 1,800 ) mathematical operations.