1
$\begingroup$

My task is multiclass classification of item to buy (next). I have a purchase history dataset with a datetime feature. From it I could engineer many new seasonality features:

  • Time of year (season, month, custom)
  • Time of month (start/mid/end, custom)
  • Time of week (each day, is weekend, custom)
  • Time of day (day part, hour, minute, custom)
  • is holiday/somehow special day, is Christmas etc.

Custom means that features do not necessarily have to be of same temporal size. As long as that feature is 'good'.

My question is: How to correctly that 'goodness' of 1/multiple features? Ideas so far:

  1. Taking a set of features and measuring accuracy metrics
  2. Taking a set of features and getting feature importance

Are those methods valid, are there more/better methods?

Also, how would I efficiently come up with an ideal combination (I assume the best encoding of real world circumstances would be a combination, not just 1) of seasonality features? Meaning, do I have to train my classifier every time for every combination I come up with, or can I dump them all in 1 training job, trusting that some combination (e.g. [is_summer, is_weekend, is_morning]) will crystallize itself with e.g. individual feature importances being higher?

Or are there some more correct/standardized processes for this kind of 'iterative feature testing'?

Also, does this depend of a type of model being used (LightGBM)?

Any answer, information or link is greatly appreciated. Thanks in advance!

$\endgroup$
1
  • $\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$
    – Community Bot
    Sep 13, 2022 at 8:26

2 Answers 2

1
$\begingroup$

Bad news:

  1. In general, feature selection is (well-known) NP-Hard problem - we never know which feature subset is best unless we exhaustively search all possible combinations, which is usually not feasible in practice.
  2. The best feature subset is dependent on the model, e.g. a linear model may perform best on a subset different from LightGBM. This comes from the fact that a (statistical) model... embodies a set of statistical assumptions, so different model may favor different features.
  3. It depends on the evaluation metric as well (obviously).

To sum up, there is no free lunch. In reality, we use a number of heuristics to do the job. You may refer to Wiki's feature selection page for an overview.

Some best practices:

  1. Try to engineer as many features as possible, and try different selection approaches.
  2. Try many models, preferably different classes (e.g. linear, tree-based, neural...)
  3. Ensemble of models always performs better than individual one.

The above points apply to machine learning problems in general, including your particular problem.

$\endgroup$
1
$\begingroup$

Depends on what you want to achieve. If you only care about performance, the straightest way to go is to build as much features as possible then dump them in a lgbm, then maybe build some ensemble with some advanced NNs. You can see this in the last Kaggle Tabular competition for exemple. (AMEX: https://www.kaggle.com/competitions/amex-default-prediction/discussion)

If you want something else (i.e. explainable model) the way to go would be to test some characteristics of features. Start with information value (package optbinning would do it - see amex comp.); in the age of Covid I would add some metrics about relationship to target to check for stability over time. Then you can select the top 100 features with no change in relationship. Logistic regression with L1 reg would help to further select the features down to <10.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.