I am using tensorflow's DNNRegressor to model a multivariate regression problem. I want to form an optimal feature subset from a mixed bag of categorical and continuous features. What would be the best way to proceed? The reason I want this approach to be independent of the model is because I couldn't find much about feature selection/evaluation in direct context of tensorflow.
There is no best way, but if there was, Tensorflow would definitely not be it. There are three ways I can think of, and each of them is worse than the others in some cases (no free lunch):
- Measure correlation of variables with the output variable, and take the variables with the highest correlation. This is a rather poor method as it considers linear correlation, and this is not a very good measure of dependence. Instead of this, although a bit more computationally intensive, is to measure the distance correlation between the output variable and all the variables, and select the features with highest distance correlation.
- Fit a linear model with L1 penalization (Lasso). Lasso will automatically select variables for you setting the weights of the non important variables to 0. The variables with the nonzero weights are the ones you can select.
- Fit a random forest or a gradient boosting, and take the variables with the highest feature importances. This has worked very well for me in practice.
I hope this helps.