0
$\begingroup$

When NN is construsted, batch size is not defined and place holder is used and its summary(tensorfow) shows the batch size as None.

This is useful because you can change batch size later.

In case of a simple model with 10 input features, 1 hidden layers with 10 neurons and output layer, the shape of the hidden layer would be (None, 10), which means if the batch size is 20, the hidden layer would have the shape of (20, 10).

When the model is used to predict for a single output, with shape(10, 1), how does the math work?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

I guess you have a confusion here. The None part represents the number of samples.

For example if you have a neural network with architecture 100-50-10, it means that you have

  • (None,100) : input layer shape
  • (100,50) : shape for weights connecting input to hidden layer
  • (None,50): shape for hidden layer given by (None,100)*(100,50) matrix multiplication
  • (None,50): shape after nonlinearity application.
  • (50,10): shape for the shape matrix between hidden and output layer
  • (None,10) : output layer shape (None,50)*(50,10) matrix multiplication

So if youre feeding a single input sample the shapes would be:

(1,100)[Input] => (1,100)(100,50) = (1,50)[Hidden Layer] => (1,50)*(50,10)=(1,10)[Output Layer]
Improve this answer
$\endgroup$
4
  • $\begingroup$ Why does the input layer shape have to be (None, 100) and not(1, 100)? $\endgroup$
    – Hiro Nakagame
    Nov 13, 2020 at 20:37
  • $\begingroup$ @HiroNakagame If you have one sample it will be (1,100), The None is a place holder for the number of samples. $\endgroup$
    – Anoop A Nair
    Nov 13, 2020 at 20:42
  • $\begingroup$ so the hidden layer has wx + b where x(input) is (None, 100) and w is (100, 50) which results in wx = (None, 50). How can that be added with b = (1, 50)? $\endgroup$
    – Hiro Nakagame
    Nov 13, 2020 at 20:52
  • $\begingroup$ the bias is added to the each of the hidden layer neurons. Since there are only 50 neurons in the hidden layer b has the shape (50,1). The addition of bias means that you're adding it to each of the rows of the (None, 50) matrix. Please go through this. missinglink.ai/guides/neural-network-concepts/… $\endgroup$
    – Anoop A Nair
    Nov 13, 2020 at 22:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.