Brute-force feature selection and cross-validation

Question

There is an existing score made of 10 parameters; each parameter is equally weighted & the total score is found by summing the score for each parameter.

I want to try to reduce the number of parameters in this score, but keep them equally weighted.

I have data on 500 people with the score & two outcomes of interest.

As the number of parameters are small, I started by doing a brute-force approach to look at all the possible combinations of parameters, and asses their performance via the AUC of the two outcomes.

What I'm worried about is that the resulting parameters may not be generalisable (i.e. I'm overfitting), so to mitigate against this I want to employ cross validation.

I think I should 10 times:

1. Randomly sort the 500 people, and split it into a test & train set.
2. Using the training set, work out the best parameters.
3. Print the AUC of the two outcomes using the best combination of parameters (from step 2) using the test set

Then, select the combination of parameters that's most frequently selected.

Does that make any sense? If there's a completely different way of doing feature selection that's best for this sort of scenario, that would also be useful!

Answer 1

I think that your method does make sense, it's indeed a kind of cross-validation and it would help obtaining a more reliable estimate of performance.

Technically I think the process that you describe is bootstrap aggregation (or bagging): repeatedly sampling (usually with replacement) and calculating the average performance on the test set. It also offers several advantages:

• you can observe the subset of parameters selected every time, which gives you an indication about the stability of the subset.
• you can also calculate a confidence interval for the performance, instead of only the mean performance.

However with this method I would recommend repeating the process more than 10 times, if possible try 100 or even 1000 times.

Note that there are various cross-validation methods available, Wikipedia has a quite good list imho.