3
$\begingroup$

I have found this image link I would like to know what NNs are not deep neural? The first three? Also what kind of functional activations do they use?

$\endgroup$

4 Answers 4

11
$\begingroup$

The term 'deep' in deep learning is not well defined, but is generally meant to apply to networks that have stacked multiple layers on top of each other to create a deep network. It is therefore up for debate when a network is considered deep or not, but you could use the definition that a deep network has two or more hidden layers. From your overview, examples of non-deep networks would therefore be the perceptron and simple feed forward network. The activations used in the network have no relation to whether a model is deep or not as it only relates to the number of layers in the network.

$\endgroup$
2
  • $\begingroup$ Thanks! What about Auto Encoder (AE), Variational AE (VAE), Denoising AE (DAE), Sparse AE (SAE) and Deep Feed Forward (DFF)? Can I consider them as Neural Network or Deep Learning? $\endgroup$
    – Inuraghe
    Mar 28, 2022 at 15:16
  • $\begingroup$ Based on the architectures shown and the definition of a deep network above the four auto encoder architectures would not be deep whereas the deep feed forward network would be. However, keep in mind that the architectures shown are a general overview of what for example an auto encoder looks like, however there are also auto encoders with more layer which would be considered deep. $\endgroup$
    – Oxbowerce
    Mar 29, 2022 at 7:20
2
$\begingroup$

The single layer neural network - the Perceptron - was first implemented by Frank Rosenblatt in 1957 using hardware (!!) - that is: resistors, electrical motor, and a potentiometer! The perceptron had one layer of neurons, (not including the output layer), and it could tell the difference between simple shapes and letters.

According to Google's N-Gram Viewer it was a long time before the term 'Deep Neural Network' was used in a book (~1980), and much longer (~ late 2000's) before it gained popular usage:

enter image description here

So to answer the question, I suspect although it's somewhat subjective, I think the Perceptron is the only true single-layer neural network in your diagram, and therefore the others would be considered 'deep'. Some may consider other 'shallow' neural networks (like 'feed forward netural nets') to not be deep, but it's worth noting that although shallow in structure, the term 'feed forward netual network' didn't come into popular parlance until slightly after the term 'deep neural network' (see here).

$\endgroup$
1
$\begingroup$

Alexnet was a revolution in image recognition in 2012 (https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf) and the paper was titled "Deep convolutional neural networks". Referring to neutral networks as "deep" was unusual at that time.

They used 8 layers, and hugely increased the state of the art (SOTA).

I conclude from that, that at least in 2012, 8 layers was considered deep.

Since then, I'd argue that 8 layers is no longer considered deep, but pretty standard for many image processing tasks.

I'd therefore argue that the definition of deep has changed and increased over time, and we've become better at training ever-deeper networks.

$\endgroup$
1
  • $\begingroup$ Are you relying solely on this article? Or have you found other information that says Deep Learning has more than 8 layers? $\endgroup$
    – Inuraghe
    Mar 31, 2022 at 7:26
0
$\begingroup$

One of the reasons deep neural networks have taken off is because GPU hardware has improved. The computations needed for practical deep networks are often highly parallelizable. Thus, I think one indicator of "deep" in addition to the standard criteria of ">= 2 layers" is that the network is structured so that many computations can be done in parallel.

Conversely, practical networks which are not highly parallelizable are often not "deep".

$\endgroup$
4
  • 1
    $\begingroup$ I think this is exactly the wrong way around. The standard parallelization in neural networks is wide, not deep. $\endgroup$
    – MSalters
    Mar 29, 2022 at 7:27
  • $\begingroup$ @MSalters What I mean is: the per-layer computations are massively parallel in both the forward and backward directions, which enables one to stack layers deeply since the cost of an individual layer is greatly reduced. $\endgroup$ Mar 29, 2022 at 7:29
  • $\begingroup$ That makes more sense, yes. $\endgroup$
    – MSalters
    Mar 29, 2022 at 7:40
  • $\begingroup$ @MateenUlhaq It's not sufficient, however, unless you can also deal with the vanishing gradient problem. $\endgroup$ Mar 29, 2022 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.