1
$\begingroup$

I used scikit.REFCV and mlxtend.SFS (backward) on the same data, same classifier, same cv, same scorer,... I also did a third version with sample weights passed to SFS's estimator

RFE

And i'm conflicted to see that many discrepencies. Should not the largest subsets have more or less same CV score, at least?

From my unterstanding, RFECV and SFS backward should work the same except the first one selects subset based on feature importance and the latter one on the metric provided.

Thanks for any input.

$\endgroup$
  • $\begingroup$ A first guess: skl splits first and uses the same split for the entire process, while mlxtend remakes the splits each time? But you say something about time-dependent CV in your github issue (github.com/rasbt/mlxtend/issues/582 ), so I'm not sure whether the splits even can be different? $\endgroup$ – Ben Reiniger Aug 31 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.