If I have training data X, with N features, and I do feature selection, and discover n of N (where n < N) are basically relevant to target data Y (the other N - n are basically irrelevant), how do I assess the impact of feature selection of a feature selected dataset, n features, to full dataset, N features?

Do I compare the performance on an identical neural network trained on "XN" vs. "Xn"? For example, if N = 20, say, and n = 5, and if I have a model with 3 layers, 18 nodes each, and an activation function F1. Do I feed the same model with 20 and 5 inputs respectively? (Comparing well a model of 3 layers, 18 nodes, active_func = F1, learns for 20 and 5 input features). "apples and apples"

Or do I, for example, do a hyper parameter optimisation in both, so that I may be testing the impact of feature selection on my ML performance, with for "XN" a model of 3 layers, 18 nodes, active_func = F1 ("the optimal hyper parameters for this dataset"), and another with 2 layers, 12 nodes, active_func = F2, ("the optimal hyperparameters for model with dataset n = 5"). "apples and oranges"

? :) Thanks!



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.