I am participating on a kaggle competition. The dataset has around 100 features and all are unknown (in terms of what actually they represent). Basically they are just numbers.
People are performing a lot of feature engineering on these features. I am wondering how exactly one is able perform feature engineering on features which are unknown? Can somebody please help me understand this and some tips on how can I perform feature engineering on unknown features?
You do not need domain knowledge (the knowledge of what your data mean) in order to do feature engineering (finding more expressive ways of framing your data).
As Tu N. explained, you can find "quick and dirty" combinations of features that could be helpful pretty easily. Given an output $y$ and an individual feature $x$, you can take the following transforms, $x' \in \{e^x, \log(x), x^2, x^3, \tanh(x)\}$. A quick check of the usefulness of the transformation is if the correlation between $\{y,x'\}$ is higher than the correlation between $\{y,x\}$.
Warning on correlation: Correlation does not show everything and depending on the model you are using (highly non-linear such as NN or RF) and interaction with other variables, a change in correlation could mean nothing.
However, if you are using a simple linear model like logistic regression, it is an OK indicator of performance. The best way to evaluate such a transformation, however, as noted by Fokhruz Zaman, would be to build a model with and without your transformed feature, and see how the validation error (on your Cross-Validation folds) evolves.
It is rather easy to spot single-feature transformations this way. Those apply to a lot of data, where a more expressive relation between your input and output could be on a different scale. For example, the relationship between Income and "Happiness" appears to be logarithmic, but you would never record the log of a participant income directly.
Finding combinations of features is more difficult. For a start, if you want to test every addition of 2 features, and you have $D$ features, you have an order of $D^2$ transformations to test. In order to find such transformations, you can apply a nonlinear model (such as NN or RF) to the problem and try to see what it is that it is learning. If you can identify what an intermediate layer in a NN is doing, you can pre-compute its result and add it as a new feature. It will not need to compute it again, and it will probably try to learn something new.
It can be hard to interpret the internal representation of a NN, or even interpret feature importance in a Random Forest. An easier, and probably more suited method for this purpose, the model would be Boosting with decision trees. There are a lot of libraries implementing Boosting, and if you are into Kaggle competition as your post seems to imply, XGBoost seems used by a lot of participants, so you might find some help/tutorials on what I'm going to describe.
First, run your boosting algorithm using only stumps, 1-level decision trees. Stumps are very weak, but Boosting makes it a reasonable model. This will act as your baseline. Depending on the library you are using, you should be able to display pretty easily which are the most used features, and you should plot them against the response (or do a histogram if the response is categorical) to identify some pattern. This might give you an intuition on what would be a good single feature transformation.
Next, run the Boosting algorithm with 2-level decision trees. This model is a lot more complex than the previous one; if two variables taken together have more power than taken individually, this model should outperform your previous one (again, not in terms of training error, but on validation error!). Based on this, you should be able to extract which variables are often used together, and this should lead you to potential multi-feature transformations.
On related material, I would advise the following videos as they are easy to follow
You can take different combinations of features such as sum of features: feat_1 + feat_2 + feat_3 ..., or product of those. Or you can transform features by log, or exponential, sigmoid ... or even discretize the numeric feature into a categorical one. It's an infinite space to explore.
Whatever combination or transformation that increases your Cross-Validation or Test Set performance then you should use it.
Could you please explain more with specific examples about your following statement?
The dataset has around 100 features and all are unknown (in terms of what actually they represent). Basically they are just numbers.
I'm not sure how can you do Feature Engineering without good understanding of your Data Set and the given attributes!!
Tabular data is described in terms of observations or instances (rows) that are made up of variables or attributes (columns). An attribute could be a feature.
"The idea of a feature, separate from an attribute, makes more sense in the context of a problem. A feature is an attribute that is useful or meaningful to your problem. It is an important part of an observation for learning about the structure of the problem that is being modeled. ... ... ...
In computer vision, an image is an observation, but a feature could be a line in the image.
In natural language processing, a document or a tweet could be an observation, and a phrase or word count could be a feature.
In speech recognition, an utterance could be an observation, but a feature might be a single word or phoneme. "
Please visit the following URL for more:
Discover Feature Engineering, How to Engineer Features and How to Get Good at It
