# Different runs of feature selection algorithm giving different set of selected feature. How to choose the best set among them?

I am using the forward feature selection algorithm from MATLAB. The code is as follows:

 X=combine_6_non;
y=target;
c = cvpartition(y,'k',10);
opts = statset('display','iter');
[fs,history] = sequentialfs(@fun,X,y,'cv',c,'options',opts)


The function fun is as follows:

function err = fun(XT,yT,Xt,yt)
model = svmtrain(XT,yT, 'Kernel_Function', 'rbf', 'boxconstraint', 1);
err = sum(svmclassify(model, Xt) ~= yt);
end


Now for different runs of the selection algorithm I am getting different feature sets. How should I zero down to the best feature set?

• Is it possible to set a seed or random_state? If yes, then that should solve the issue :) Aug 5, 2016 at 8:18
• @Dawny33 what are these things? Aug 5, 2016 at 8:23
• Some algorithms are (partly) stochastic, thus randomly choosing value to explore the space of solutions. However, computers cannot generate random numbers but pseudo-random numbers that are simulated based on an initial seed. If you fix it, you should get the same results each time. There is no randomness in the SVM, thus I would guess it is in the splitting part Aug 5, 2016 at 8:38
• @Dawny33 so the partitioning is leading to this right? Aug 5, 2016 at 8:40
• @Rishika Yes. So, pl read up on how to set a random_state or a seed in Matlab, similar to Python :) Aug 5, 2016 at 8:47

You can set a seed or a random state for the splitting process. This helps in generating a fixed random number everytime, which helps you get the same data everytime you do the CV split.

This can be done by set_seed(...) in R, and adding the random_state = ... in relevant functions parameters in Python. So, please add the relevant parameter for Matlab, and your CV would do good.

Setting the seed will not entirely solve the problem of obtaining different optimal feature sets. It will result in an arbitrarily optimal one. The feature selection algorithm obtaining different optimal feature sets for different cv-fold setups or initialisations, implies that your ML problem is sensitive to the data and hyperparameters.

Ideally you should increasing your model capacity or the number of examples until the optimal feature sets are stable between runs. If that's not an option or does not improve the issue the conservative approach would be to use the intersection of these feature sets, or less conservatively, the union.