Best practices for selecting categorical features

Question

I'm trying to create a classifier that will predict whether someone will attend an interview or not. Each data point is for a single candidate and contains details such as the location of the interview, the candidate's current location, the job skill requirements, interview type etc. All of the data is categorical.

There are also some features, which you may not think would have an impact on the candidate's attendance e.g. marital status. My initial thought was, based on "common sense" (I use this term loosely), to drop this feature, but I wanted to take a more rigorous approach to determine the importance of this. However, I am unsure of what is best practice. How do I go about determining whether this is a feature that can be removed or not?

Secondly, is there a downside to keeping it even if it has limited predictive power? Aside from creating a more cumbersome model, could it adversely impact the accuracy of a prediction?

Thanks.

Answer 1

For removing categorical variable, generally used techniques is chi-square distribution based selection. The technique falls under general umbrella term univariate feature selection.

Answer 2

It is really good to have the intuition about the model & features. But practically it is recommended to follow feature selection procedures.

However, I am unsure of what is best practice. How do I go about determining whether this is a feature that can be removed or not?

A limited common sense or intuition can be applied here, we will obviously know "mobile number" of the candidate is of no use (It has less predicting power). But we may not know whether mode of travel of the candidate will have its effect on interview presence (Based on the location & commutability).

It is necessary to feature engineer (running feature selection algorithms) your data to find out what are some effective features. Look here for the methods involved. I found tree based selection algos worked well for me. But it really depends on the data.

Secondly, is there a downside to keeping it even if it has limited predictive power? Aside from creating a more cumbersome model, could it adversely impact the accuracy of a prediction?

It is always better to avoid features which are of less/no use to improve the model accuracy unless you have very minimal no of features which is effective. And it also helps you to train your model faster because of dimensionality reduction. Refer

Scikit-learn doc for examples

Answer 3

With labels, you mainly encounter two problems with treed based algorithms that can overfit in both cases:

• Rare values
• Cardinality

Rare values are labels/ categories within a categorical variable that are only present for a small percentage of the observations. There is no rule of thumb to determine how small is a small percentage, but typically, any value below 5% may cause over-fitting in trees.

High Cardinality: One approach that is heavily used in Kaggle competitions, is to replace each label of the categorical variable by the count, this is the number of times each label appears in the dataset.

There's a great course on Feature Engineering on Udemy. Dr. Soledad Galli goes into much detail of rare values and cardinality.

In particular, she investigates different approaches to a Kaggle dataset from Mercedes Benz. Pretty interesting and advanced.

https://www.udemy.com/feature-engineering-for-machine-learning/learn/v4/overview