Is (manual) feature extraction outdated?

Question

I recently attended a PhD thesis defence in which one committee members claimed that "manual feature extraction is outdated. Nowadays, we have [deep] machine learning models doing that job for us automatically."

Is this statement true? If yes, please provide a reference substantiating this claim.

Edit: Apparently, there seem to be different answers depending on the data type. Thus, please let me know about any references substantiating your claims for images, time series, etc... separately.

Answer 1

In the general case, this is by no means true. Let's break down the case for different data scenarios:

• For discriminative image models (e.g. image classification/labeling) this is true for some scenarios. You just throw some convnets (even pretrained models) at your data, and that's it. Nevertheless, convnets themselves profit from the "expert knowledge" that information locality is important and so is hierarchical information processing. For some other scenarios, applying domain knowledge (e.g. specific data transformations) may give the edge to reach the needed level of quality in the results.
• For many image processing problems, neural networks work best when infused with some kind of inductive bias, e.g. attention.
• For Natural Language Processing (NLP) problems, a good amount of craftsmanship is needed nowadays, especially in the data preprocessing stage.
• For "typical data science" problems, it is also crucial to do feature extraction. You can have a look at Kaggle competitions to verify this.
• For time series problems, it is also normal to rely on expert knowledge to understand which models fit best based on the nature of the data.

However, I think that the trend of the areas where deep learning is applicable (i.e. tons of available data) is to try to devise systems that are trained end-to-end, with the least possible ad hoc processing. Nevertheless, many times this is achieved by infusing the expert knowledge into the network in the form of inductive biases.

Answer 2

No, manual feature extraction is not outdated. In addition, manual feature extraction is hard to do-away, given, a data scientist needs business and domain logic to build a robust model to replicate and capture trend and pattern from data. Nevertheless, there are exceptions such as image data.

Depends, if its image data, yes the statement is true. There are many deep learning techniques e.g. CNN which extract features automatically. However, if your data is structures i.e. standard table format, one will need to use p_value, correlation analysis, chi-test, and feature_selection models such as PCA and dimensionality-reduction to select features.

Here are a list of feature extraction techniques (i.e.manual feature extraction techniques, requiring human intervention; these are not deep learning extraction techniques, though automated.):

• Independent component analysis
• Isomap
• Kernel PCA
• Latent semantic analysis
• Partial least squares
• Principal component analysis
• Multifactor dimensionality reduction
• Nonlinear dimensionality reduction
• Multilinear Principal Component Analysis
• Multilinear subspace learning
• Semidefinite embedding
• Autoencoder

Below is a list of deep-learning feature extraction techniques:

• Convolution

Answer 3

One of the main strengths of DL methods is that they can work from raw data, and often perform better this way than traditional methods with carefully crafted features. So it's indeed very tempting to consider traditional feature engineering as obsolete, since it requires more work and often results in lower performance.

However one should be careful before discarding feature engineering in this way:

• First, as scientists we should be wary of the dynamic nature of technological trends. For example very few ML experts would have bet on neural nets 15-20 years ago as the next big thing. We should take stock of the evolution of ML methods, not blindly adopt the latest technology.
• DL methods are computationally expensive and usually require a large amount of data. There are still plenty of applications/problems where more lightweight traditional methods are a better fit.
• DL methods are by nature less open to interpreting their results. Interpretability/explainability is already an important issue and is likely to become even more important as applications of ML meet real-life problems: ethical issues (what if a ML system is racist?), legal issues (why did a ML system make a bad decision and who is responsible?). By contrast, some statistical methods such as decision trees offer a very clear explanation of their decisions.
• In some cases leaving feature engineering to DL is suboptimal. There have been a few results (in NLP as far as I'm aware) showing that on some specific problems carefully crafted features performs better than DL. I don't know if these are significant or just exceptions to the rule. Subjective interpretation: there might be a risk of "design laziness", i.e. counting on DL to do the job instead of properly understanding and structuring the problem.

Answer 4

Transfer learning

A different perspective on this is to take a look at what exactly has (in some fields) replaced feature engineering, and that is usually not simply having neural networks learn to extract features from your task-specific data, but it usually involves transfer learning from large quantities of somewhat similar data, which can be labeled for a different task or unlabeled at all.

In general, all the models do need good features; however, in some domains of data - for example, photos of objects and faces, and natural language text analysis - the features that you can get from large sources of non-task-specific data (using e.g. a good ImageNet model for photos, or good contextual word embeddings for text) are quite good, and contain most of the things that you might include in manual feature extraction.

So the argument is that if for your particular problem it is possible to apply such a transfer learning approach, then that is mandatory (since the knowledge that you can "import" outweighs anything you can realistically do to augment your small task-specific dataset) but manual feature engineering is useful but optional. It's likely to be helpful and bring some improvement, but not that much. If in "pre-NN" methods the results before feature engineering were horrible and you needed to do significant feature work to get good results, then nowadays simply picking the first reasonable, best practice data representation will get you most of the way to the best results possible. You need to ensure that you don't make major mistakes that either throw away important data or include an unrealistic signal factor (e.g. "detect skin cancer from images" where all the positive images have a ruler next to the lesion), but putting a lot of effort in feature engineering in most domains will give just marginal improvements over a small but skilled effort to ensure that the data representation is reasonable.