2
$\begingroup$

I am working on a prediction (binary classification) problem

Currently I get an AUC score of 85-86 and F1-score of 81

Questions

1) The above performance is based on 6 well-known features

2) Let's say I add another feature and I see my AUC and F1 score improve by 1 or 2 points? So is this the only way to know that it really helps/value addition to the model?

3) How do I justify that these features really help in predicting the output?

4) Is there anyway to prove or validate that adding this feature really helps my model and improves the outcome? Is it only using changes in AUC score?

5) For instance, I can add multiple features and my AUC increases by few decimal points, so can I say that they are important or useful or drive predictions? Yes ofcourse, but adding 10 features (where 6 features really impact the outcome and rest 4 increase the auc only by few decimal points). doesn't really overfit. Am I right? Because my prediction score doesn't go beyond 86

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Cross Validation

You had a post where we discussed causality, but with ML models assumption is that data represents your problem entirely and has all the information in it. In other words every pattern that you pickup in your train data you can expect it to behave pretty similar in production, hence with this assumption what you want is to "evaluate the entire train" (thats what you can do with CV.) and if it score good on average over all folds you want to add this change (be it a new feature)

Improve this answer
$\endgroup$
7
  • $\begingroup$ Hi, thanks upvoted. Yes, I understand. what I am trying to know is how do I say that addition of feature is useful? Is it only through improvement in my metric like F1 score or AUC score or is there any other approach to validate the usefulness of a feature?. If my metric goes up, I don't have to worry about anything and can claim that this new feature helps the model better. Am I right to understand this? $\endgroup$
    – The Great
    Commented Dec 27, 2019 at 9:46
  • 1
    $\begingroup$ Or is it by what you suggested yesterday through causal approaches like CFE and PFI? $\endgroup$
    – The Great
    Commented Dec 27, 2019 at 9:50
  • $\begingroup$ PFI is individual featue estimator. I would rephrase the question you asked. Can it happen that f1 score was increased with a feature (on all of the folds !) but it turned out feature was useless on holdout/production? only if the covariate shift was too big. So under normal assumptions CV and bigger f1 movement is your safest bet to test if you should add this feature. $\endgroup$
    – Noah Weber
    Commented Dec 27, 2019 at 10:01
  • $\begingroup$ Hi,can I break this into simple terms as few terms are new to me. Yes, currently I have split my data into train and test. I don't really capture my metrics in train data. But yes, I use feature selection approaches and CV to get the best parameters and estimator. Once I get the model with best parameters, I just pass the test data to this which gives me the above metrics $\endgroup$
    – The Great
    Commented Dec 27, 2019 at 10:06
  • $\begingroup$ Now I am not sure which additional features to retain and drop. I mean adding few features increases the F1-score by decimal points or sometimes even by 1 whole point. So whether it is useful to add these features for this slight improvement. As my model AUC is only at 86, adding features (2-3 FEATURES) to take it to 87 doesn't really make the model complex. Am I right? $\endgroup$
    – The Great
    Commented Dec 27, 2019 at 10:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.