3
$\begingroup$

One of the managers at my company asked if there is a I could include a metric demonstrates that the my model is not using too many Features/Variables. Is there a metric or best practice that does this? Do you have a good way of communicating this to the business?

Specifically, she asked if I could include AIC about my models. After I asked some questions though, this is not quite what management is looking for because, unless I'm mistaken, AIC is only useful for comparing models with the same features.

In any case, I was able to use train-test validation for the model I was presenting, so AIC would have been overkill. To deal with this problem in practice I create a baseline model which uses all the variables, and then pair down as many variables as I can without sacrificing too much performance (on the train and test sets).

Improve this question
$\endgroup$
2
  • $\begingroup$ I dont understand why you say AIC is not suitable here (besides, I think there is not a good understanding how AIC, or any similar metric, works). $\endgroup$
    – Newbie
    Nov 4, 2020 at 15:25
  • $\begingroup$ @Newbie, I decided against AIC because I use a train-test-out of sample validation strategy, so it would not have added any value. You could probably use it in a manner described. But, add stated it would have been overkill. $\endgroup$
    – D. Kelleher
    Nov 6, 2020 at 14:17

2 Answers 2

Reset to default
1
$\begingroup$

My approach would be quite similar to yours with a baseline, maybe just a bit more general:

Assuming it's practical (i.e. the training process is not too costly), you could train/test multiple models with various number of features, e.g. 10%, 20%...,100% of the features. For every subset you use an appropriate feature selection method, preferably something like genetic algorithm but it might be too costly. If it's not possible a simple individual feature selection method, e.g. using information gain, but it's likely not to be optimal. Once all the models have been trained, tested and evaluated, a plot of the performance as a function of the number of features should (hopefully) show a curve which increases less and less but never reaches the plateau. If it does reach the plateau before your chosen number of features, it means that you could actually cut down the number features.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I like this as a way to visualise the process, since the issue is how best to explain my process. $\endgroup$
    – D. Kelleher
    Oct 27, 2020 at 12:31
0
$\begingroup$

AIC is a perfectly suitable metric here. It is not for models that have the same features, (because what would the difference be between such models?). It is for comparing models that use different features, and penalises those that use more features.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.