2
$\begingroup$

I received a dataset for analysis that had ~100 numeric columns with anonymous column names(X1, X2, X3, etc...) and asked to do a binary classification. My resulting classification algorithm using a SVM had good accuracy (> 95%), but I was unable to do much in the way of feature engineering or feature generation other than the standard scaling, null-value replacement, etc, since I had no intuition about the columns.

Is there any standard logic as to how to do some sort of automated feature generation, i.e. some simple mathematical combinations of various columns to create new, useful features? Does this sort of thing have any mathematical basis for linear or tree-based models? Or is feature engineering only really meaningful when one has intuition based on the column names...

$\endgroup$
  • $\begingroup$ You can try analyzing the feature importance of the cols and then try to use statistics and decipher the meaning of the underlying cols $\endgroup$ – Aditya Jul 30 at 5:35
  • 1
    $\begingroup$ Maybe relevant to your question orges-leka.de/automatic_feature_engineering.html The method is based on Bourgain Embedding. $\endgroup$ – orgesleka Aug 31 at 15:42
0
$\begingroup$

I think your question has multiple answers. Let's start with a tool created with this purpose named featuretools. You can read more about it on the excellent documentation, but just to give you an intuition, think about a dataset that is on transaction level, that is, you have one row a new transaction, and then you want to make predictions based on customer level, you will create a series of features based on transactions, for example:

  • mean, max, min, sum, std of transactions values
  • number of transactions
  • number of unique products bought
  • ...

This tool will automate all these tasks based on entity sets and relationships between your datasets.

Now, think about your situation, you might use PolynomialFeatures from scikit-learn to generate interactions and polynomial features that is more automated than other approaches.

Having said that, based on your score, in a real application, using these approaches, or trying to improve performance using more features, is overwhelming and not useful. We use kaggle as our standard training, and I think it is really good, but business requires explanation as well, and they will be happy with less predictive algorithm if they can trust on it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.