I have been working for a while in credit problems for classification and regression and on these problems I have had the necessity of improving already good performing models, for this when apparently almost domain knowledge features have been included, I have used some non-common sense features that actually worked for example:

Entropy of the binned distribution of payment history
Linear/quadratic trend of balance over time
Cosine of due balance


Do you have other examples of features that seemed "totally crazy" but that at the end improved your model's performance?

If so, Is there any theoretical reason that/those feature(s) worked in your problem?

  • 1
    $\begingroup$ The examples you mention don't seem crazy at all, they convey information which is useful for the prediction. Feature engineering is not about trying "crazy" things, it's about presenting available information in an optimal way for the learning method. The key is expert knowledge, not random features. Btw by trying many random features you're bound to overfit your model, and if you test all of them with the same data you're not going to notice it. $\endgroup$
    – Erwan
    Dec 9 '20 at 19:51
  • 1
    $\begingroup$ Just a thing, how are you suppose to overfit when trying random features (this is not what I intended to ask but, features that unexpectedly worked without evident or easy explanation) If the feature simply does not work, I do not see how you can overfit (at most that feature would add no information) Without taking into account the case in which you use a metric prone to "improving" when adding more features like R2. do you have a formal statement on that? "...and if you test all of them with the same data you're not going to notice it" This will simply happen regardless of the features. $\endgroup$ Dec 9 '20 at 22:05
  • $\begingroup$ Sorry I wasn't clear: I meant that if one tries a large number of features (like in the examples you mentioned by trying various functions on some relevant indicators, so not completely random) then eliminates all the bad ones and keeps the ones which improve performance, then it's possible that the selected features perform better by chance on the dataset (overfitting). Of course this can be checked by keeping some instances as test set and running the model based on these selected features on these fresh instances, exactly like when evaluating after tuning some hyper-parameters. $\endgroup$
    – Erwan
    Dec 9 '20 at 22:25