8
$\begingroup$

I am going over the Udacity tutorial on Neural Networks.

Here's a diagram from the tutorial:

enter image description here

What makes this a '2 layer neural network'?

  • I was under the impression that the first layer, the actual input, should be considered a layer and included in the count.

  • This screenshot shows 2 matrix multiplies and 1 layer of ReLu's. To me this looks like 3 layers. There are arrows pointing from one to another, indicating they are separate. Include the input layer, and this looks like a 4 layer NN.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

Input layer is a layer, it's not wrong to say that.

However, when calculating the depth of a deep neural network, we only consider the layers that have tunable weights.

Improve this answer
$\endgroup$
7
  • $\begingroup$ So in the picture above that I posted, I see 2 matrix multiplies and a ReLu. Isn't that 3 layers? $\endgroup$
    – Monica Heddneck
    Commented Sep 16, 2016 at 5:39
  • $\begingroup$ ReLU is just element-wise activation, not a weighted layer. $\endgroup$
    – nn0p
    Commented Sep 16, 2016 at 7:07
  • $\begingroup$ @MonicaHeddneck nn0p is correct. Don't be too concern what's exactly drawn in the picture. Focus on the actual layer architecture (the text below). $\endgroup$
    – SmallChess
    Commented Sep 16, 2016 at 7:18
  • $\begingroup$ ohhhhh so ReLu's can be layers because they are not parametric! This is exactly what I was wondering about. $\endgroup$
    – Monica Heddneck
    Commented Sep 16, 2016 at 7:18
  • 1
    $\begingroup$ @MonicaHeddneck ReLus is part of the hidden layer. The picture you show is misleading. $\endgroup$
    – SmallChess
    Commented Sep 16, 2016 at 7:21
9
$\begingroup$

This is a 2-layer network because it has a single hidden layer and an output layer. We don't count the first layer. When we say 3 layers, we actually mean 2 hidden layers and an output layer. Perhaps this helps you?

EDIT: We don't count the input layer because there's no parameter (bias + weights). In actual implementation, it's not implemented. Netural network framework simply "connects" the input features to the first adjustable layer (eg: hidden layer).

enter image description here

Improve this answer
$\endgroup$
5
  • $\begingroup$ That makes a lot of sense. Can you take a look at the image I posted -- I think I'm seeing 3 layers that all are parameterized. $\endgroup$
    – Monica Heddneck
    Commented Sep 16, 2016 at 5:49
  • 1
    $\begingroup$ @MonicaHeddneck I don't know what exactly the picture is drawing, but it's not important. The text below tells you exactly why it's a two-layer network. $\endgroup$
    – SmallChess
    Commented Sep 16, 2016 at 5:59
  • $\begingroup$ I agree now...the text is correct and the picture is misleading. When I asked this question, I wasn't sure if the picture was correct and the text was misleading. Thanks for your help. $\endgroup$
    – Monica Heddneck
    Commented Sep 16, 2016 at 7:28
  • 3
    $\begingroup$ @SmallChess Hi, can you please give the URL of the article/webpage from which you imported the picture containing the two NN diagrams and the Naming convertions: paragraph ? $\endgroup$
    – SebMa
    Commented Jul 20, 2018 at 13:55
  • $\begingroup$ @SebMa I googled partial of the text, luckily google is smart enough :) cs231n.github.io/neural-networks-1 $\endgroup$
    – Yana Agun Siswanto
    Commented Jul 10, 2019 at 23:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.