# How to perform feature engineering on unknown features?

I am participating on a kaggle competition. The dataset has around 100 features and all are unknown (in terms of what actually they represent). Basically they are just numbers.

People are performing a lot of feature engineering on these features. I am wondering how exactly one is able perform feature engineering on features which are unknown? Can somebody please help me understand this and some tips on how can I perform feature engineering on unknown features?

You do not need domain knowledge (the knowledge of what your data mean) in order to do feature engineering (finding more expressive ways of framing your data).

As Tu N. explained, you can find "quick and dirty" combinations of features that could be helpful pretty easily. Given an output $y$ and an individual feature $x$, you can take the following transforms, $x' \in \{e^x, \log(x), x^2, x^3, \tanh(x)\}$. A quick check of the usefulness of the transformation is if the correlation between $\{y,x'\}$ is higher than the correlation between $\{y,x\}$.

Warning on correlation: Correlation does not show everything and depending on the model you are using (highly non-linear such as NN or RF) and interaction with other variables, a change in correlation could mean nothing.

However, if you are using a simple linear model like logistic regression, it is an OK indicator of perfomance. The best way to evaluate such a transformation, however, as noted by Fokhruz Zaman, would be to build a model with and without your transformed feature, and see how the validation error (on your Cross-Validation folds) evolves.

It is rather easy to spot single-feature transformations this way. Those apply to a lot of data, where a more expressive relation between your input and output could be on a different scale. For exemple the relationship between Income and "Happiness" appears to be logarithmic, but you would never record the log of a participant income directly.

Finding combinations of feature is more difficult. For a start, if you want to test every addition of 2 features, and you have $D$ features, you have an order of $D^2$ transformations to test. In order to find such transformations, you can apply a nonlinear model (such as NN or RF) to the problem and try to see what it is that it is learning. If you can identify what an intermediate layer in a NN is doing, you can pre-compute its result and add it as a new feature. It will not need to compute it again, and it will probably try to learn something new.

It can be hard to interpret the internal representation of a NN, or even interpret feature importance in a Random Forest. An easier, and probably more suited method for this purpose, model would be Boosting with decision trees. There are a lot of libraries implementing Boosting, and if you are into Kaggle competition as your post seem to imply, XGBoost seems used by a lot of participants, so you might find some help/tutorials on what I'm going to describe.

First, run your boosting algorithm using only stumps, 1-level decision trees. Stumps are very weak, but Boosting makes it a reasonnable model. This will act as your baseline. Depending on the library you are using, you should be able to display pretty easily which are the most used features, and you should plot them against the response (or do an histogram if the response is categorical) to identify some pattern. This might give you an intuition on what would be a good single feature transformation.

Next, run the Boosting algorithm with 2-level decision trees. This model is a lot more complex than the previous one; if two variables taken together have more power than taken individually, this model should outperform your previous one (again, not in term of training error, but on validation error!). Based on this, you should be able to extract which variable are often used together, and this should lead you to potential multi-feature transformations.

On related material, I would advise the following videos as they are easy to follow

You can take different combinations of features such as sum of features: feat_1 + feat_2 + feat_3 ..., or product of those. Or you can transform features by log, or exponential, sigmoid ... or even discretize the numeric feature into a categorical one. It's an infinite space to explore.

Whatever combination or transformation that increases your Cross-Validation or Test Set performance then you should use it.

• I would take a bit of issue with "Whatever combination or transformation that increases your Cross-Validation or Test Set performance then you should use it.". Blindly trying stuff until something improves your performance metric could result in discovering a relationship that doesn't make sense and causes overfitting. This could really hurt performance on new observations down the road. – Hersheezy Aug 26 '16 at 16:50

The dataset has around 100 features and all are unknown (in terms of what actually they represent). Basically they are just numbers.


I'm not sure how can you do Feature Engineering without good understanding of your Data Set and the given attributes!!

Tabular data is described in terms of observations or instances (rows) that are made up of variables or attributes (columns). An attribute could be a feature.

"The idea of a feature, separate from an attribute, makes more sense in the context of a problem. A feature is an attribute that is useful or meaningful to your problem. It is an important part of an observation for learning about the structure of the problem that is being modeled. ... ... ...

In computer vision, an image is an observation, but a feature could be a line in the image.

In natural language processing, a document or a tweet could be an observation, and a phrase or word count could be a feature.

In speech recognition, an utterance could be an observation, but a feature might be a single word or phoneme. "

Please visit the following URL for more:

Discover Feature Engineering, How to Engineer Features and How to Get Good at It

• The attributes in the OP's case have been anonymised to the point where domain knowledge cannot be used. This is quite common in Kaggle competitions, yet limited forms of feature engineering are still possible. Typically new features are created in bulk then tested, as opposed to using subject knowledge to guide their generation. An example: kaggle.com/c/bnp-paribas-cardif-claims-management/data – Neil Slater Mar 11 '16 at 10:04
• as Neil said, the features are anonymized, that is what i mean from my that statement. – user2409011 Mar 11 '16 at 10:07