5
$\begingroup$

I am going over the Udacity tutorial on Neural Networks.

Here's a diagram from the tutorial:

enter image description here

What makes this a '2 layer neural network'?

  • I was under the impression that the first layer, the actual input, should be considered a layer and included in the count.

  • This screenshot shows 2 matrix multiplies and 1 layer of ReLu's. To me this looks like 3 layers. There are arrows pointing from one to another, indicating they are separate. Include the input layer, and this looks like a 4 layer NN.

$\endgroup$
2
$\begingroup$

Input layer is a layer, it's not wrong to say that.

However, when calculating the depth of a deep neural network, we only consider the layers that have tunable weights.

$\endgroup$
  • $\begingroup$ So in the picture above that I posted, I see 2 matrix multiplies and a ReLu. Isn't that 3 layers? $\endgroup$ – Monica Heddneck Sep 16 '16 at 5:39
  • $\begingroup$ ReLU is just element-wise activation, not a weighted layer. $\endgroup$ – nn0p Sep 16 '16 at 7:07
  • $\begingroup$ @MonicaHeddneck nn0p is correct. Don't be too concern what's exactly drawn in the picture. Focus on the actual layer architecture (the text below). $\endgroup$ – HelloWorld Sep 16 '16 at 7:18
  • $\begingroup$ ohhhhh so ReLu's can be layers because they are not parametric! This is exactly what I was wondering about. $\endgroup$ – Monica Heddneck Sep 16 '16 at 7:18
  • 1
    $\begingroup$ @MonicaHeddneck ReLus is part of the hidden layer. The picture you show is misleading. $\endgroup$ – HelloWorld Sep 16 '16 at 7:21
5
$\begingroup$

This is a 2-layer network because it has a single hidden layer and an output layer. We don't count the first layer. When we say 3 layers, we actually mean 2 hidden layers and an output layer. Perhaps this helps you?

EDIT: We don't count the input layer because there's no parameter (bias + weights). In actual implementation, it's not implemented. Netural network framework simply "connects" the input features to the first adjustable layer (eg: hidden layer).

enter image description here

$\endgroup$
  • $\begingroup$ That makes a lot of sense. Can you take a look at the image I posted -- I think I'm seeing 3 layers that all are parameterized. $\endgroup$ – Monica Heddneck Sep 16 '16 at 5:49
  • 1
    $\begingroup$ @MonicaHeddneck I don't know what exactly the picture is drawing, but it's not important. The text below tells you exactly why it's a two-layer network. $\endgroup$ – HelloWorld Sep 16 '16 at 5:59
  • $\begingroup$ I agree now...the text is correct and the picture is misleading. When I asked this question, I wasn't sure if the picture was correct and the text was misleading. Thanks for your help. $\endgroup$ – Monica Heddneck Sep 16 '16 at 7:28
  • 3
    $\begingroup$ @SmallChess Hi, can you please give the URL of the article/webpage from which you imported the picture containing the two NN diagrams and the Naming convertions: paragraph ? $\endgroup$ – SebMa Jul 20 '18 at 13:55
  • $\begingroup$ @SebMa I googled partial of the text, luckily google is smart enough :) cs231n.github.io/neural-networks-1 $\endgroup$ – Yana Agun Siswanto Jul 10 '19 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.