Is it possible that out of several attributes $p$, only one attribute could be selected by a model in the feature selection and training phase? Then basically we are fitting a line. Basically, I was thinking that one should select the model with lesser feature set over a model with large number of feature. If one feature ($p =1$) gives better performance then should I not select that particular feature only?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
You could have a look at an R package called mboost (documentation), which performs standard boosting (fitting a linear model using some features you give it) and the performs a coefficient update for only the feature that contributes to the largest reduction in error.
All coefficients start at zero, so after many iterations, this results in some coefficient with large values, some with small coefficients and normally some with coefficients equal to zero... they were not selected at all. This means you have inherent feature selection during training.
Check out the tutorial paper, which is very helpful in getting started.
Here is an image, which shows the coefficient development during training:
it shows the names of the features on the right... you can see that some values are still at zero once training has finished.
The package has built in functionality for cross-validation, plotting and so on.
EDIT:
You can think of the training process as follows:
- run a regression on the data
- measure which feature was best able to fit (e.g. had smallest error)
- this feature "won the round" and gets its coefficient in the final equation increased by an amount (e.g. 0.001)
- repeat steps 1-3 until a threshold/criterion is met
- All features that didn't win a single round can be removed
How many features do you have? If it isn't too many, you can simply run the model many times, adding/removing a single feature each time. You could also try using some metrics such as BIC (Bayesian Information Criterion) to decide which model explains the data best with the given features.
-
$\begingroup$ Thank you for your answer and the link to the paper. I will go through it. But to summarize the key findings if it is found that 1 attribute gives good performance then it is legal to go forward with it?I am only using a SVM classifier and perhaps then try with MLP. $\endgroup$ Commented Jul 10, 2018 at 17:47
-
$\begingroup$ The key finding is that some features don't contribute as much as others. See my edits. $\endgroup$– n1k31t4Commented Jul 10, 2018 at 18:00