I received a dataset for analysis that had ~100 numeric columns with anonymous column names($X1$, $X2$, $X3$, etc...) and asked to do a binary classification. My resulting classification algorithm using a SVM had good accuracy (> 95%), but I was unable to do much in the way of feature engineering or feature generation other than the standard scaling, null-value replacement, etc, since I had no intuition about the columns.

Is there any standard logic as to how to do some sort of automated feature generation, i.e. some simple mathematical combinations of various columns to create new, useful features? Does this sort of thing have any mathematical basis for linear or tree-based models? Or is feature engineering only really meaningful when one has intuition based on the column names...

  • $\begingroup$ You can try analyzing the feature importance of the cols and then try to use statistics and decipher the meaning of the underlying cols $\endgroup$ – Aditya Jul 30 '19 at 5:35
  • 1
    $\begingroup$ Maybe relevant to your question orges-leka.de/automatic_feature_engineering.html The method is based on Bourgain Embedding. $\endgroup$ – orgesleka Aug 31 '19 at 15:42
  • $\begingroup$ 95% accuracy sounds good, but what's your class size distribution? $\endgroup$ – Itamar Mushkin Jun 30 at 14:23

I think your question has multiple answers. Let's start with a tool created with this purpose named featuretools. You can read more about it on the excellent documentation, but just to give you an intuition, think about a dataset that is on transaction level, that is, you have one row a new transaction, and then you want to make predictions based on customer level, you will create a series of features based on transactions, for example:

  • mean, max, min, sum, std of transactions values
  • number of transactions
  • number of unique products bought
  • ...

This tool will automate all these tasks based on entity sets and relationships between your datasets.

Now, think about your situation, you might use PolynomialFeatures from scikit-learn to generate interactions and polynomial features that is more automated than other approaches.

Having said that, based on your score, in a real application, using these approaches, or trying to improve performance using more features, is overwhelming and not useful. We use kaggle as our standard training, and I think it is really good, but business requires explanation as well, and they will be happy with less predictive algorithm if they can trust on it.

| improve this answer | |
  • $\begingroup$ Explanation is hard with anonymous columns. $\endgroup$ – Itamar Mushkin Jun 30 at 14:24
  • $\begingroup$ What do you mean by that? @ItamarMushkin $\endgroup$ – Victor Oliveira Jun 30 at 14:31

Another related problem, which I think worth considering, before generating more features, is to determine which columns are important for classification task, i.e. improve prediction of target variable.

One common way is to rely on feature importance scores, but a disadvantage of this is that the scores are only available after training model. To guess which features improve prediction of target variable before training, we can compute correlation among features and target. Note that correlation computation only works for numeric features. To measure relationship between a categorical feature and a numeric target, we need other measurements, called association. You can find out more in this nice article.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.