Very conceptual question:
I have a TensorFlow model that works well. I have isolated about 70 features worth of data and when training, my validation accuracy stays around 32% while my training accuracy is up to 89%! However, when I use only 24 of those 70 features, I get a validation accuracy of 38% and a training accuracy of 53%. I am using regularization, heavy dropout, and early stopping.
Is this a good problem to have? Should I keep the 70 features while boosting overfitting parameters? What should I do about this? Maybe it's a good idea to keep using only the 24 best features?
Is it generally O.K. to shed some features by just discarding them? Definitely some emotional attachment to my features here, but is it common for an ML process to simply discard these features as opposed to working with them to achieve higher accuracy?