Automatic Feature Engineering

Question

I came across a Machine Learning software which I won't name, that claims that completely automates the Feature Engineering process from any source without domain knowledge, implying pretty much that turns Data Scientists unnecessary for this step. Now I find that claim a bit suspicious since it goes against the common sense of what I learned Feature Engineering consists of.

But considering that it might be a lack of knowledge from my part to what point we can automate the featuring engineering process? I can think of some general approaches like statistical methods (median,etc..) or several types of coding (binary, polynomial,etc..) for categorical variables, but could anyone give me an overview state of the art approach regarding this subject or point me in the right direction?

Answer 1

In my experience, when people claim to have an automated approach to feature engineering, they really mean "feature generation", and what they're actually talking about is that they've built a deep neural network of some sort. To be fair, in a limited sense, this could be a true claim. Properly trained deep neural networks can handle any number of pairwise correlations between individual features or groups of features. That said, without a great deal of up-front data pre-processing tools that know how to intelligently handle different types of input data (e.g., free text, images, etc.), none of this would be possible. Bottom line, it takes a great deal of manual effort to do something automatically.

Answer 2

There are several methodologies to do that... For the tool you have been talking about (which I will not mention it's name with respect to you) all sorts of pre-coded functions are run at once, it's basically a vast rule engine.

1) Easiest method is to run mini-trees with a random combination of variables (sounds like random forest)... Each tree that has some significant predictive power on classification is a variable and the nodes are it's categories...

2) You can build auto-encoders (https://en.wikipedia.org/wiki/Autoencoder), it's easy to do but hard to interpret, basically what deep learning does is automatic feature engineering, that's why it takes so much time to compute ;)

3) You can do symbolic regression (https://en.wikipedia.org/wiki/Symbolic_regression), possibly using a genetic optimization algorithm to select variables and some mathematical operators to come up with a nice formula that has some power of classification. So when you have some data on a company balance sheet it provides many formulas like EBITDA...

Answer 3

Well, there is some serious research going on in this direction under the label of "feature learning". But as far as I know, it's not mature enough yet to be packaged into a software tool that renders manual software engineering superfluous.

But there are important successes being achieved in that direction. Modern image recognition with deep neural networks often relies on features that the deeper layers of the neural network compute themselves instead of hand-crafted features developed by humans. So it's not entirely science-fiction. If you want to learn more about the image recognition part of the topic, you may want to watch this beautiful video lecture by Andrew Ng, one of the leading researchers in this field.

Answer 4

Have a look at h2o's driverless AI or featuretools.py, I don't believe either use deep learning, but rather apply an array of mathematical functions on your data set and then performs Feature Synthesis/merging features. The auto-generated feature set is then iteratively tested against your model and discarded or improved.

So yes, while you need domain knowledge/experts to realise that adding or creating new data could make a huge impact, for exploratory purposes you could definitely use auto-features.

Answer 5

Almost anything can be automated but that doesn't mean it makes sense theoretically.

Developing features takes insight and content knowledge from the field that you are studying in order to make sense. This is why even the best data scientists need to work with content experts to make theoretically sound models

Answer 6

Maybe relevant for this question:

http://www.orges-leka.de/automatic_feature_engineering.html

The method is based on Bourgain Embedding and works whenever one has a distance between two data points. But to have a "good" distance solving the job at hand, one needs domain knowledge as there are many distances around for the same data type.