what are the differences between feature weighting and feature selection? And is feature importance like feature weighting?
Feature selection is a pre-processing step.
Feature weighting is a learned step.
This is a pre-processing step. Meaning that you choose which features the model even gets to see. This is important because in large enterprises, not all data is available from the same source or there is a cost to getting certain data. Therefore, features needed to be selected beforehand and effort is put to retrieve only the necessary data.
- Medical Records might be useful for determining insurance premiums, but due to regulations you might leave that feature out.
- You are detecting topics of incoming e-mails. The past purchases of the sender might help to determine the topic in the e-mail but accessing the purchases database is more difficult than the e-mails, so you might choose to use only the e-mail texts instead.
This is a learning step. At this point, you can assume that feature selection is done, and that you have access to all the data you wanted.
Now the idea is to determine the importance of the feature coming to your model. This can be manually set, but ideally is based on some learned metric.
Feature importance is like feature weighting.
- You have an ensemble model, where all the feature coming into this model are actually predictions from other models. You might weight the predictions of these other models based on their individual performance. Then, your ensembler takes predictions from good performing models with more weight than from those with poorer individual performance.