Which one first: algorithm benchmarking, feature selection, parameter tuning?

Question

When trying to do e.g. a classification, my approach currently is to

1. try out various algorithm first and benchmark them
2. perform feature selection on the best algorithm from 1 above
3. tune the parameters using the selected features and algorithm

However, I often cannot convince myself that there may be a better algorithm then the selected one, if the other algorithms have been optimized with the best parameter / most suitable features. At the same time, doing a search across all algorithms * parameters * features are just too time-consuming.

Any suggestion on the right approach / sequence?

Answer 1

I assume you mean feature selection as feature engineering. The process I usually follow and I see some people do is

1. Feature engineering

2. Try a few algorithms, usually highly performant ones such as RandomForest, Gradient Boosted Trees, Neutral Networks, or SVM on the features.

   2.1 Do simple parameter tuning such as grid search on a small range of parameters

If the result of step 2 is not satisfactory, go back to step 1 to generate more features, or remove redundant features and keep the best ones, people usually call this feature selection. If running out of ideas for new features, try more algorithms.

If the result is alright or close to what you want, then move to step 3

3. Extensive parameter tuning

The reason for doing this is that classification is all about feature engineering, and unless you know some incredible powerful classifier such as deep learning customized for a particular problem, such as Computer Vision. Generating good features is the key. Choosing a classifier is important but not crucial. All the classifiers mentioned above are quite comparable in terms of performance, and most of the time, best classifier turns out to be one of them.

Parameter tuning can boost performance, in some cases, quite a lot. But without good features, tuning doesn't help much. Keep in mind, you always have time for parameter tuning. Also, there's no point of tuning parameter extensively then you discover a new feature and redo the whole thing.

Answer 2

I just saw your question. It is COMPLETELY WRONG to do feature selection first and then tune the model using cross-validation. In elements of statistical learning and this blog post, it is clearly mentioned that:

What has happened? The problem is the predictors have an unfair advantage, as they were chosen in step (1) on the basis of all of the samples. Leaving samples out after the variables have been selected does not correctly mimic the application of the classifier to a completely independent test set, since these predictors "have already seen" the left-out samples.

CV methods are unbiased only if all your model building is done inside the CV loop. So do feature selection inside the CV loop for parameter tuning. It can be done easily using the filter wrapper in the MLR package in R.

Answer 3

I have already answered a similar question here. The process would be:

• Transformation and Reduction: Involves processes like transformations, mean and median scaling, etc.
• Feature Selection: This can be done in a lot of ways like threshold selection, subset selection, etc.
• Designing predictive model: Design the predictive model on the training data depending on the features you have at hand.
• Cross Validation and parameter tuning:
• Final Prediction, Validation

Always try and do feature engineering before model selection. Then, select the model according to the best features (or the features which tend to influence the problem/dependant variable better.)

Answer 4

If you are prepared to spend time learning how a new tool works, you can try Auto-Sklearn. It does all that's needed to build an ML pipeline for you.

Feature preprocessing, selection, model ensemble building and tuning through cross-validation. Depending on the amount of data you have, it may or may not be a faster way to a good prediction, but it is certainly a promising one.