Background: I am working on a binary classification of health insurance claims. The data I am working with has approximately 1 million rows and a mix of numeric features and categorical features (all of which are nominal discrete). The issue I am facing is that several of my categorical features have high cardinality with many values that are very uncommon or unique. I have plotted 8 of my categorical features below which had the highest counts of unique factor levels:enter image description here

Alternative to Dummy Variables: I have been reading up on feature hashing and understand that this method is an alternative that can be used for a fast and space-efficient way of vectorizing features and is particularity suitable for categorical data with high cardinality. I plan to utilize Scikit Learn's FeatureHasher to perform feature hashing on my categorical features with more than 100 unique feature levels (I will create dummy variables for the remaining categorical features with less than 100 unique feature levels). Before I implement this I have a few questions relating to feature hashing and how it relates to model performance in machine learning:

  1. What is the primary advantage of using feature hashing as opposed to dummying only the most frequently occuring feature levels? I assume there is less information loss with the feature hashing approach but need more clarification on what advantages hashing provides in machine learning algorithms when dealing with high cardinality.

  2. I am struggling to understand how to best determine n_features in Scikit Learn's FeatureHasher. Clearly higher hashing dimensions will encode more information and provide better model predictions at the cost of computationally more expensive training. When researching FeatureHasher I was looking at it as a method to reduce the dimension of the encoded table generated (I have several categorical features with well over 1,000 unique levels) while preserving information by not dropping uncommon or unique features levels. With that being said, how can best determine the optimal n_features for each column in my dataset with the goal of obtaining an encoded table that does not exponentially increase the number of features being fed into my classification models?

  3. I am interested in evaluating feature importance after evaluating a few separate classification models. Is there a way to evaluate hashed features in the context of how they relate to the original categorical levels? Is it possible to reverse hashes or does feature hashing inevitably lead to loss of model interpretability?

Sorry for the long post and questions. Any feedback/recommendations would be much appreciated!

  • $\begingroup$ I am not here to answer your question per se, but saying have tried out CatBoost gradient boosting (catboost.ai/docs/concepts/python-installation.html)? It automatically encode your cat. features using multiple methods, and you get to see the feature importanc. Often their default parameter does a good job compared with other models, and less trouble for dealing cat. variables. $\endgroup$ – TwinPenguins Apr 3 '20 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.