2
$\begingroup$

According to the documentation - CRAN

Boruta is an all relevant feature selection wrapper algorithm, capable of working with any classification method that output variable importance measure (VIM); by default, Boruta uses Random Forest.

Source: Boruta documentation

The package has one parameter getImp that defines the importance to be used, which by default is random forests from the Ranger package. So theoretically one could use xgboost's xgb.importance() function as the source of feature importance but I cannot find an example of how to use such parameter in practice because so far I've been unlucky. Is it possible at all to do this with the Boruta package or do I have implement from scratch the Boruta algorithm again using xgboost? And if it's possible how?

$\endgroup$
6
$\begingroup$

Boruta is by universal reputation dog-slow and not very good. Boruta runs take many hours or days. VIF feature-selection algorithm is not objective, anyway. You can program your own feature-selection that runs faster. I ran Boruta a few times on various datasets and it wasted 4 days of my time, and the result was inconclusive.

Here's the fast-and-dirty not-100%-scientific approach, culled from consensus in a lot of Kaggle competitions and suchlike. It's 10-1000x faster than Boruta and probably more accurate.

  1. Run some fast exploratory trees (RF/XGB), i.e. not very deep, not too many trees. Extract feature importances. Repeat for several random seeds and look at average or absmax feature importances.
  2. For now, throw out very-high-cardinality features (e.g. user-ID, zipcode etc.), since trees won't tend to split on them until very deep and small nodes, so initially exclude them. Let's define "Very-high-cardinality" as >= ~0.3 distinct values per record, or less in a larger dataset.
  3. Throw out all near-zero features, zero- and very-low-importance features (< 0.02-0.05 f.i. or lower is a rule of thumb).
  4. Measure the matrix of which features are highly correlated (< 0.4) to each other. Corrplot is a nice way to visualize these, and order correlated features near each other). Pick a sensible color-scale. Identify subgroups of correlated features. Arbitrarily select one feature from those groups (you can revisit this selection later). Or you can decide if this warrants using PCA.
  5. Rerun, your CV accuracy should have gone way up, and your training time will be way faster (since tree training is quadratic to no. of features). Revisit choices of borderline-low-important or correlated feature, and again use multiple random seeds.
  6. For exploratory, 5-fold CV is fine, but if training time is an issue, 3-fold works. 10-fold is unnecessary.
  7. For the subset of remaining features, use a 'respectable' method. See the excellent posts on CrossValidated in feature-selection.

Somebody with a PhD in Stats will probably wag their finger at me for this :)

| improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ For anyone who comes across this response many years later; 1) Boruta is much faster these days; it switched to the ranger package instead of the randomForest package in R and has been built with scikit learn in Python (it is also used in Kaggle competitions quite frequently now). 2) Sub setting using feature importance (or any feature selection methods involving your response variable y) outside of a cross validation scheme is ill-advised and will almost certainly lead to overfitting and an overly optimistic estimation of predictive performance. The variable importance lists... $\endgroup$ – aranglol Apr 11 '19 at 2:22
  • $\begingroup$ as explained above are created using the entire data set and therefore incorporates information that exists within each test fold in the cross validation process of step 5. On Kaggle, there exists a held out test set, usually a subset of the actual test set that final scores are calculated on, that users can upload predictions for up to five times a day that somewhat gets around this problem. $\endgroup$ – aranglol Apr 11 '19 at 2:29
  • $\begingroup$ Thanks for the updates @aranglol. Useful to mention package version numbers, if you can. $\endgroup$ – smci Apr 11 '19 at 9:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.