Cannot understand feature extraction

Question

I'm following an AI course and we've just entered the deep learning chapter. Speaking about the difference between classic machine learning models and deep learning, it turns out one of the most relevant points in favor of a neural network is that it doesn't need the feature extraction phase. That's because it can implicit learn what are the most important features directly on data we give it as input. Conversely, a machine learning system heavily relies on feature extraction before it can start training.

I've found some useful questions about this topic on this site, but still I'm a bit confused in understanding what feature extraction actually is. So I've asked my teacher to provide me an example.

He gave me this example: suppose we have a huge dataset of images and we want to classify them based on the presence or absence of at least one tree in the picture. So, an image will be classified as 1 if it contains a tree, -1 otherwise.

• in the case of a neural network, we simply feed the images to the net with their labels. That's all. The network will learn what a tree is and will optimize its weights in order to detect them
• on the contrary, if we want to use a machine learning algorithm, we also have to tell the model what a tree actually is, in addition to the labeled images. This part is the feature extraction phase

I don't understand how can I tell to a model what a tree is inside an image. I thought discover it is part of the task of the machine learning algorithm.

Answer 1

Preliminary note

as far as I know, people may use the term feature extraction in slightly different ways:

• referring to automatic methods used for dimensionality reduction which involve transforming the features (and not only selecting a subset of features).
• referring to the general process of designing and engineering the features before training/testing a model on these features.

Please be aware that I'm not 100% sure that the terminology I use myself is standard. (but I don't think it matters for the question you're asking).

Answer

Used in the context you mention, the term certainly refers to the general process of feature engineering (2nd point above).

In traditional ML models the model directly uses the features exactly as they are provided to the training algorithm. For example, a decision tree classification model tries to find the best conditions in the features to discriminate the data. This means that the algorithm selects a single feature and directly uses its range of values to build a condition, something like this:

 if color == 'green' then go to subtree 1 else go to subtree 2

Thus the type of information that can be used by a model is strictly limited by the algorithm and the features which are provided to it: there's no way to consider any information which is not directly available as a feature. As a consequence the stage of feature engineering is crucial. For example in text classification there are a lot of choices to make: provide the words frequency or TFIDF as features? Provide all the word or remove stop words? Lemmatize or not? Add POS tags? And these are just some very standard questions, specific tasks may require a very detailed analysis. Importantly, it's not possible to just provide all the features one can think of because this would almost certainly cause the model to overfit due to the high number of features. Generally in this stage the goal is to give the model the best and most precise clues that will help it make correct predictions, and also avoid giving it too many irrelevant features to prevent overfitting.

That's the stage that DL doesn't need: thanks to a much more complex architecture and algorithmic process, a DL model can itself select and combine the features it needs to perform well.

I thought discover it is part of the task of the machine learning algorithm.

Sure, but in traditional ML in order for the model to discover the most important patterns for a specific task, the features have to be prepared in a way which maximizes the chances of the model to find these. Simile: if a teacher tells their students to revise page 63 of the textbook for the test, it's likely that the students will perform better than if they have no clue about the topic and need to revise the whole textbook.