Machine Learning vs Deep Learning

Question

I am a bit confused by the difference between the terms "Machine Learning" and "Deep Learning". I have Googled it and read many articles, but it is still not very clear to me.

A known definition of Machine Learning by Tom Mitchell is:

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.

If I take an image classification problem of classifying dogs and cats as my taks T, from this definition I understand that if I would give a ML algorithm a bunch of images of dogs and cats (experience E), the ML algorithm could learn how to distinguish a new image as being either a dog or cat (provided the performance measure P is well defined).

Then comes Deep Learning. I understand that Deep Learning is part of Machine Learning, and that the above definition holds. The performance at task T improves with experience E. All fine till now.

This blog states that there is a difference between Machine Learning and Deep Learning. The difference according to Adil is that in (Traditional) Machine Learning the features have to be hand-crafted, whereas in Deep Learning the features are learned. The following figures clarify his statement.

I am confused by the fact that in (Traditional) Machine Learning the features have to be hand-crafted. From the above definition by Tom Mitchell, I would think that these features would be learned from experience E and performance P. What could otherwise be learned in Machine Learning?

In Deep Learning I understand that from experience you learn the features and how they relate to each other to improve the performance. Could I conclude that in Machine Learning features have to be hand-crafted and what is learned is the combination of features? Or am I missing something else?

Answer 1

In addition to what Himanshu Rai said, Deep learning is a subfield which involves the use of neural networks.These neural networks try to learn the underlying distribution by modifying the weights between the layers. Now, consider the case of image recognition using deep learning: a neural network model is divided among layers, these layers are connected by links called weights, as the training process begins, these layers adjust the weights such that each layer tries to detect some feature and help the next layer for its processing.The key point to note is we don't explicitly tell the layer to learn to detect edges, or eyes, nose or faces.The model learns to do that itself.Unlike classical machine learning models.

Answer 2

As a research area, Deep Learning is really just a sub-field of Machine Learning as Machine Learning is a sub-field of Artificial Intelligence.

1) Unsupervised Feature Learning

Conceptually, the first main difference between "traditional" (or "shallow") Machine Learning and Deep Learning is Unsupervised Feature Learning.

As you already know, successfully training a "traditional" Machine Learning model (ex: SVM, XGBoost...) is only possible after suitable pre-processing and judicious feature extraction to select meaningful information from the data. That is, good feature vectors contain features distinctive between data points with different labels and consistent among data points with the same label. Feature Engineering is thus the process of manual feature selection from experts. This is a very important but tedious taks to perform!

Unsupervised Feature Learning is a process where the model itself selects features automatically through training. The topology of a Neural Network organized in layers connected to each other have the nice property of mapping a low-level representation of the data to a higher-level representation. Through training, the network can thus "decide" what part of the data matters and what part of the data doesn't. This is particularly interesting in Computer Vision or Natural Language Processing where it is quite hard to manually select or engineer robust features.

(picture credits: Tony Beltramelli)

As an example, let's assume we want to classify cat pictures. Using a Deep Neural Net, we can feed in the raw pixel values that will be mapped to a set of weights by the first layer, then these weights will be mapped to other weights by the second layer, until the last layer allows some weights to be mapped to numbers representing your problem. (ex: in this case the probability of the picture containing a cat)

Even though Deep Neural Networks can perform Unsupervised Feature Learning, it doesn't prevent you from doing Feature Engineering yourself to better represent your problem. Unsupervised Feature Learning, Feature Extraction, and Feature Engineering are not mutually exclusive!

Sources:

• http://deeplearning.stanford.edu/tutorial/
• https://arxiv.org/abs/1404.7828
• https://arxiv.org/abs/1512.05616 (Chapter 2, Section 2)

2) Linear Separability

Deep Neural Networks can solve some non-linearly separable problems by bending the feature space such that features become linearly separable. Once again, this is possible thanks to the network topology organized in layers mapping inputs to new representations of the data.

(picture credits: Christopher Olah)

Sources: http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/

3) Statistical Invariance

Lastly, Deep Neural Networks are surpassing traditional Machine Learning algorithms in some domains because some architectures are showcasing Statistical Invariance (ex: Spacial Statistical Invariance with Convolutional Neural Networks and Temporal Statistical Invariance with Recurrent Neural Networks)

Check this Udacity video for more details: https://www.youtube.com/watch?v=5PH2Vot-tD4

Answer 3

Inspired by Einstein, "“If you can't explain it to a six year old, you don't understand it yourself.”

All of the above answers are very well explained but if one is looking for an easy to remember, abstract difference, here is the best one I know:

The key difference is Machine Learning only digests data, while Deep Learning can generate and enhance data. It is not only predictive but also generative.

Source. Of course there is much more to it but for beginners it can get way too confusing.

Answer 4

While both machine learning and deep learning involve training models to make predictions or decisions based on data, deep learning excels at automatically learning hierarchical representations of complex patterns or features from raw data.

This can be best illustrated with an MNIST example. Imagine we are trying to classify images of handwritten digits. In traditional machine learning, we might need to manually select relevant features, like the curvature of lines or the presence of loops. However, deep learning algorithms can automatically learn these features by stacking multiple layers of interconnected neurons. Each layer extracts progressively higher-level features, such as edges, corners, and shapes, leading to a more accurate representation of the input data.

This ability to automatically learn hierarchical representations makes deep learning particularly powerful for tasks like image recognition, natural language processing, and speech recognition. It eliminates the need for manual feature engineering, making it more scalable and adaptable to a wide range of problems.

So, while both machine learning and deep learning involve training models, deep learning's ability to learn complex features automatically gives it an edge in handling intricate and unstructured data, making it a popular choice for solving challenging problems in various domains.

Answer 5

Okay, think of it like this. In machine learning algirithms, such as linear regression or random forest you give the algorithms a set of features and the target and then it tries to minimize the cost function, so no it doesnt learn any new features, it just learns the weights. Now when you come to deep learning, you have atleast one, (almost always more) hidden layer with a set number of units, these are the features that are being talked about. So a deep learning algorithm doesnt just learn the sets of weights, in that process it also learns the values for hidden units which are complex high level features of the trivial data that you have given. Hence while practicing vanilla machine learning a lot of expertise lies in your ability to engineer features because the algorithm isnt learning any by itself. I hope I answered your question.