5
$\begingroup$

While it is obviously clear that features can be ranked on basis of importance and many machine learning books give examples of random forests on how to do so, its not very clear on which occasions one should do so.

In particular, for boosting methods, is there any reason why one should do feature selection. Wouldn't the boosting methods themselves eliminate the low importance feature.

Isn't is just always better to add more features (if one didn't have the practical problem of time limitations).

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

There is a difference between boosting and features selection. It is very important to understand that the original boosting algorithm or bagging algorithm have been modified and augmented with many features selection and/or data sampling ( over/ down/ synthetic) to improve the accuracy. Let us talk about the difference between bagging and boosting : Booth of them are random subspace based algorithms, the difference is in bagging we used uniform distribution and all the samples have the same weight , in boosting we use non- uniform distribution, during the training the distribution will be modified and difficult samples will have higher probability. The second difference is the voting. In bagging it is average voting , in boosting it is an weighted voting.

Features selection algorithms try to find the best set of features that can separate the classes. But there is no explicit consideration for difficult or easy samples and what is the used training algorithm. In boosting, the algorithm selects the feature that minimize the error , the error is the sum of prob "weights" of samples that miss classified, since the difficult samples have higher weights , the selected feature will be the one that better distinguish between the difficult samples.

FE ( Features, data) --> feature set Boosting ( features, data, base learners type, initial distribution, difficult samples) --> feature set

Improve this answer
$\endgroup$
2
  • $\begingroup$ I didn't understand from your answer if you want to say feature selection should or should not be performed before using a boosted learning model. A somewhat related question is whether one should use as large a number of new features (for example take many different powers of continuous features) as allowed by computational constraints. $\endgroup$
    – Borun Chowdhury
    Commented Apr 7, 2016 at 18:18
  • $\begingroup$ It depends on the boosting algorithm you are using. If you are using simple boosting algorithm and you have a very large number of features it is a good idea to use feature selection , but if you are using boosting algorithm that includes feature selection ( ex viola jones object detection adaboost algorithm) no need to perform the features selection , it is already included $\endgroup$
    – Bashar Haddad
    Commented Apr 7, 2016 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.