Amount of multiplications in a neural network model

Question

I'm currently reading this book and want someone to tell me if what currently I'm assuming about neural network is right or not. https://www.pyimagesearch.com/deep-learning-computer-vision-python-book/

If a layer has 30 neurons: In a feedforward meural network each neuron of the first layer multiplied with all the neurons of the second layer. That would be 30 neurons of the first layer multiplied by the 30 neurons of the second layer that would give a total of 900 (30*30 = 900) multiplications (is this is correct?)

But those 900 multiplications is only for the first layer and the second layer of the neural network(nn). So if I have a feedforward nn that has 3 layers I would have to do 900 more multiplications because they are the multiplications of the output of the first layer (that they are the input of the second layer) with the weight of the third layer.

So to recap what I said so far I have a feedforward nn with 3 layer with 30 neurons in the first two layers multiply each neuron with all the neurons of the second layer it would be 30 * 30 = 900 multiplications for each additional layer that I add, it adds 900 additional multiplications.

Then for a model with three (3) fully connected layer would be 900 multiplications in the first two layer because for what I said earlier and 900 multiplications in the second and third layer for a total of 1,800 (900 + 900) multiplications excluding an activation function and this is only for a forward pass, is my understanding correct?

And in addition to a forward pass in a typical Neural network they also have a backward pass that because of my calculations doing earlier they are 1,800 derivatives (gradient) for the entire backward pass. Am I correct assuming this for a Neural network?. That's why a CPU computer takes so long to train a model because it has to do about 3,600 (1,800 + 1,800 ) mathematical operations.

Answer 1

Essentially you are correct, there are a lot of calculations necessary to process inputs and train neural networks.

You have some terminology a bit wrong or vague. E.g.

In a feedforward meural network each neuron of the first layer multiplied with all the neurons of the second layer.

The neurons do not multiply together directly. A common way to write the equation for a neural network layer, calling input layer values $x_i$ and first hidden layer values $a_j$, where there are N inputs might be

$$a_j = f( b_j + \sum_{i=1}^{N} W_{ij}x_{i})$$

where $f()$ is the activation function $b_j$ is the bias term, $W_{ij}$ is the weight connecting $a_j$ to $x_i$.

So if you have $M$ neurons in the hidden layer, you have $N\times M$ multiplications and $M$ separate sums/additions over $N+1$ terms, and $M$ applications of the transfer function $f()$

And in addition to a forward pass in a typical Neural network they also have a backward pass that because of my calculations doing earlier they are 1,800 derivatives (gradient) for the entire backward pass.

It doesn't work quite so directly, and there as a small factor of more calculations involved (you do not calculate each derivative with a single multiplication, often there are a few, some results are re-used, and other operations may be involved). However yes you do need to calculate a derivative for each weight and bias term, and there are roughly that number of weights in your network that require the calculations done.

Your suggested numbers are actually quite small compared to typical neural networks used for image problems. These typically perform millions of computations for a forward pass.

That's why a CPU computer takes so long to train a model because it has to do about 3,600 (1,800 + 1,800 ) mathematical operations.

Actually that is a trivial number of calculations for a modern CPU, and would be done in less than a millisecond. But multiply this out by a few factors:

• You must do this for each and every example in the training data

• Your example network is small, think bigger

• This does not include the activation function calculations - typically slower than a multiply

• Your rough estimate ignores some of the necessary operations, so as a guesstimate, multiply number of CPU-level operations by 3 or 4 from your analysis.

  . . . and the number of operations does start to get to values where CPUs can take hours or days to perform training tasks in practice.