What are the exact differences between Deep Learning, Deep Neural Networks, Artificial Neural Networks and further terms?

Question

After having read some theory I am getting a bit confused about the following terms:

• Deep Learning
• Deep Neural Network
• Artificial Neural Network
• Feedforward Neural Network

So, what seems clear to me is that Deep Neural Networks are Artificial Neural Networks with multiple layers (usually more than 1 hidden layer). However I have read several times that:

"Deep Neural Networks are feedforward Neural Networks with many layers."

I know what a feedforward Neural Network is, but to my understanding Deep Neural Networks is a term for ALL Artificial Neural Networks with multiple layers between input and output layer? Shouldn't there also be for instance Deep Recurrent Neural Networks? Is it correct that Deep Neural Networks must feedforward Neural Networks? This would in turn mean that Deep Recurrent Neural Networks cannot be referred to as Deep Neural Networks.

Moreover, I see there is a large variety of deep learning architectures, such as:

• Convolutional Neural Networks

• Residual Neural Networks

• Deep Belief Networks

• Deep Boltzmann machines

• ...

However, now also Wikipedia gives me a hard time to distinguish all the terms, saying: "Deep learning architectures such as deep neural networks, deep belief networks, recurrent neural networks and convolutional neural networks have been applied to fields including..."

So, my concrete questions rising from all the thoughts above are:

1. Does the term Deep Neural Networks ONLY belong to feedforward Neural Networks?

2. If the answer to 1) is yes: is the Wikipedia definition correct as you can read it up there? This would mean that i.e. a Convolutional Neural Network with multiple layers must be referred to as a Deep Convolutional Neural Network, which is not a subclass of Deep Neural Networks?

3. Is the term Deep Neural Networks a collective term for ALL Artificial Neural Networks with multiple layers, or just for all feedforward Neural Networks with multiple layers?

4. Would it be more accurate to use the collective term "Deep Learning Architectures" with strictly separated subclasses as Wikipedia suggests?

Answer 1

I will try to explain it in the simplest way I can-

Deep Learning - Deep learning is a machine learning technique that teaches computers to do what comes naturally to humans: learn by example. It is a subclass of Machine Learning where (in layman terms) we use only Neural Networks. https://www.mathworks.com/discovery/deep-learning.html

Deep Neural Network (DNN) - This is just a term for a neural network with many layers and many nodes in layers, which is not usually found in a shallow network. There is no clear boundary between a shallow and deep neural network. I suggest you watch Andrew Ng's course on deep learning to get a better understanding.

Artificial Neural Network (ANN) - This is just another term for a neural network, all neural networks are ANN's.

Feedforward Neural Network - It is a type of neural network where there is no feedback connections. In technical terms, the information flows only in one direction (input to output) in the forward propagation stage. https://towardsdatascience.com/deep-learning-feedforward-neural-network-26a6705dbdc7

Shouldn't there also be for instance Deep Recurrent Neural Networks?

There is no need for Deep Recurrent Neural Networks, it would simply be Recurrent Neural Networks.

Is it correct that Deep Neural Networks must feedforward Neural Networks?

No, RNN is a type of Deep Neural Network which is not a feedforward NN.

So, to answer your questions,

1. Does the term Deep Neural Networks ONLY belong to feedforward Neural Networks?

No, the term Deep Neural Networks belong to all Neural Networks with multiple layers. (Architecture does not matter).

3. Is the term Deep Neural Networks a collective term for ALL Artificial Neural Networks with multiple layers, or just for all feedforward Neural Networks with multiple layers?

Yes, it is a collective term for ALL Artificial Neural Networks with multiple layers.

4. Would it be more accurate to use the collective term "Deep Learning Architectures" with strictly separated subclasses as Wikipedia suggests?

Deep learning architectures just refer to the different neural network architectures used for different tasks - CNN for image processing, RNN for sequence (text, audio etc) processing etc. So, when you are doing a particular task, you can use the associated architecture name. For general purpose, you can just use the term DNN.

Hope this clears your confusion.