What will happen if almost constant values for features?

Question

In a problem of an epidemiology dataset, is it desirable to keep the features that have almost constant values? For example, In case of the feature, type_of_residence Large for 97 percent and Small for 2.7 percent of subjects. Is it okay to keep this feature?

My target variable is outcome of patients and this data set is unbalanced. Like oversampling and under sampling techniques in class imbalance problem, is there any process in ML to address other predicting features?

I am not doing feature selection using ML at this point for knowing the importance of this feature. But would like to know is there any general rules regarding keeping or not keeping these kind of features in dataset.

Answer 1

If a feature is truly constant across all the instances, then it's useless for classification and it can be removed.

But in any other case, as in your example, a feature can be useful even if constant with most instances. In your example it might turn out that the target class is strongly correlated with type_of_residence, so it would be a mistake to get rid of it since your model can take advantage of it.

Unless you have a good reason to get rid of a feature (expert knowledge), keep all of them. If you must reduce the number of features (for instance to avoid overfitting), then the selection should be based on information about which features are the least informative (e.g. using some correlation measure).

Ideally the best way to know which features are useful is to try to train a classifier and evaluate it with every possible subset of features (but it's rarely possible in practice)