1
$\begingroup$

Very conceptual question:

I have a TensorFlow model that works well. I have isolated about 70 features worth of data and when training, my validation accuracy stays around 32% while my training accuracy is up to 89%! However, when I use only 24 of those 70 features, I get a validation accuracy of 38% and a training accuracy of 53%. I am using regularization, heavy dropout, and early stopping.

Is this a good problem to have? Should I keep the 70 features while boosting overfitting parameters? What should I do about this? Maybe it's a good idea to keep using only the 24 best features?

Is it generally O.K. to shed some features by just discarding them? Definitely some emotional attachment to my features here, but is it common for an ML process to simply discard these features as opposed to working with them to achieve higher accuracy?

Improve this question
$\endgroup$
11
  • 2
    $\begingroup$ You got a higher accuracy with fewer features. What's not to like? Overfitting is bad by definition. $\endgroup$
    – Emre
    Aug 21, 2017 at 0:27
  • $\begingroup$ True! My doubt stems from knowing that more features gives me really high training accuracy, so I'm not sure if I can work with more features to achieve higher test accuracy, or if I should just settle for the 24 features. $\endgroup$
    – Landmaster
    Aug 21, 2017 at 0:52
  • 1
    $\begingroup$ Maybe you can do better than 32% while retaining the other features by using more weight decay, or other form of regularization. I can not say without seeing the data. But at the end of the day, the validation set performance must decide. $\endgroup$
    – Emre
    Aug 21, 2017 at 1:00
  • $\begingroup$ If you were to see the data (maybe I could post some here?) what would you be looking for? $\endgroup$
    – Landmaster
    Aug 21, 2017 at 1:05
  • $\begingroup$ The optimal hyperparameters. Unfortunately I don't have the time to do it myself, but someone else might. $\endgroup$
    – Emre
    Aug 21, 2017 at 1:13

1 Answer 1

Reset to default
1
$\begingroup$

Generally, you should pick the model that has the highest performance on the validation dataset.

The number of features is a hyperparameter and should be tuned during the model fitting process.

Improve this answer
$\endgroup$
1
  • $\begingroup$ It is also possible that the data might not fit properly. I mean to say that algorithm not able to adjust parameter so we may need to see other model also. $\endgroup$
    – Gaurav Koradiya
    Jul 13, 2020 at 4:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.