I am using the forward feature selection algorithm from MATLAB. The code is as follows:

 c = cvpartition(y,'k',10);
 opts = statset('display','iter');
 [fs,history] = sequentialfs(@fun,X,y,'cv',c,'options',opts)

The function fun is as follows:

function err = fun(XT,yT,Xt,yt)
  model = svmtrain(XT,yT, 'Kernel_Function', 'rbf', 'boxconstraint', 1);
  err = sum(svmclassify(model, Xt) ~= yt);

Now for different runs of the selection algorithm I am getting different feature sets. How should I zero down to the best feature set?

  • 1
    $\begingroup$ Is it possible to set a seed or random_state? If yes, then that should solve the issue :) $\endgroup$ – Dawny33 Aug 5 '16 at 8:18
  • $\begingroup$ @Dawny33 what are these things? $\endgroup$ – girl101 Aug 5 '16 at 8:23
  • 1
    $\begingroup$ Some algorithms are (partly) stochastic, thus randomly choosing value to explore the space of solutions. However, computers cannot generate random numbers but pseudo-random numbers that are simulated based on an initial seed. If you fix it, you should get the same results each time. There is no randomness in the SVM, thus I would guess it is in the splitting part $\endgroup$ – Igor OA Aug 5 '16 at 8:38
  • $\begingroup$ @Dawny33 so the partitioning is leading to this right? $\endgroup$ – girl101 Aug 5 '16 at 8:40
  • $\begingroup$ @Rishika Yes. So, pl read up on how to set a random_state or a seed in Matlab, similar to Python :) $\endgroup$ – Dawny33 Aug 5 '16 at 8:47

You can set a seed or a random state for the splitting process. This helps in generating a fixed random number everytime, which helps you get the same data everytime you do the CV split.

This can be done by set_seed(...) in R, and adding the random_state = ... in relevant functions parameters in Python. So, please add the relevant parameter for Matlab, and your CV would do good.


Setting the seed will not entirely solve the problem of obtaining different optimal feature sets. It will result in an arbitrarily optimal one. The feature selection algorithm obtaining different optimal feature sets for different cv-fold setups or initialisations, implies that your ML problem is sensitive to the data and hyperparameters.

Ideally you should increasing your model capacity or the number of examples until the optimal feature sets are stable between runs. If that's not an option or does not improve the issue the conservative approach would be to use the intersection of these feature sets, or less conservatively, the union.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.