There is an existing score made of 10 parameters; each parameter is equally weighted & the total score is found by summing the score for each parameter.
I want to try to reduce the number of parameters in this score, but keep them equally weighted.
I have data on 500 people with the score & two outcomes of interest.
As the number of parameters are small, I started by doing a brute-force approach to look at all the possible combinations of parameters, and asses their performance via the AUC of the two outcomes.
What I'm worried about is that the resulting parameters may not be generalisable (i.e. I'm overfitting), so to mitigate against this I want to employ cross validation.
I think I should 10 times:
- Randomly sort the 500 people, and split it into a test & train set.
- Using the training set, work out the best parameters.
- Print the AUC of the two outcomes using the best combination of parameters (from step 2) using the test set
Then, select the combination of parameters that's most frequently selected.
Does that make any sense? If there's a completely different way of doing feature selection that's best for this sort of scenario, that would also be useful!