Different runs of feature selection algorithm giving different set of selected feature. How to choose the best set among them?

Question

I am using the forward feature selection algorithm from MATLAB. The code is as follows:

 X=combine_6_non;
 y=target;
 c = cvpartition(y,'k',10);
 opts = statset('display','iter');
 [fs,history] = sequentialfs(@fun,X,y,'cv',c,'options',opts)

The function fun is as follows:

function err = fun(XT,yT,Xt,yt)
  model = svmtrain(XT,yT, 'Kernel_Function', 'rbf', 'boxconstraint', 1);
  err = sum(svmclassify(model, Xt) ~= yt);
end

Now for different runs of the selection algorithm I am getting different feature sets. How should I zero down to the best feature set?

Answer 1

You can set a seed or a random state for the splitting process. This helps in generating a fixed random number everytime, which helps you get the same data everytime you do the CV split.

This can be done by set_seed(...) in R, and adding the random_state = ... in relevant functions parameters in Python. So, please add the relevant parameter for Matlab, and your CV would do good.

Answer 2

Setting the seed will not entirely solve the problem of obtaining different optimal feature sets. It will result in an arbitrarily optimal one. The feature selection algorithm obtaining different optimal feature sets for different cv-fold setups or initialisations, implies that your ML problem is sensitive to the data and hyperparameters.

Ideally you should increasing your model capacity or the number of examples until the optimal feature sets are stable between runs. If that's not an option or does not improve the issue the conservative approach would be to use the intersection of these feature sets, or less conservatively, the union.