I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as would be expected in many cases of description data:
in>> df[['size description', 'weight Description', 'height Description']].nunique() out>> size Description 4066 weight Description 736 height Description 3173 dtype: int64
some examples of these values could be:
Product Product Description --------- ------------------------ Ball Round bouncy toy for kids Bat Stick that kids use to hit a ball Go-Kart red/black Small motorized vehicle for kids Go-Kart blue/green Small motorized vehicle for kids Wrench Tool for tightening or loosening bolts Ratchet Tool for tightening or loosening bolts Reclining arm-chair Cushioned seat for lounging
I think that the descriptions are standardized if they fall within a particular category but at this time I cannot confirm if the number of unique descriptions are finite. At this time, my assumption would be to treat these as nominal-categorical, as these are literally descriptive and not qualitative.
To that end, my question is what are some best practices for handling categorical features such as these?
Things I have considered:
Label encoding is obviously not viable in this situation, as the descriptions have no hierarchy.
One-hot encoding seems an unlikely solution as it balloons the shape of the dataset from (55300 , 6) to (55300 , 65223) due to the high cardinality of the other variables. However, I tried it anyway and generated 98% accuracy on my test set but very poor results on an out-of-sample validation set (5k records, 0-5% accuracy). Seems pretty clear it's over-fitting and thus, not viable.
Hashing, for whatever reason, will not apply to one of the columns, but I suppose it could still be viable. I just need to figure out why it's not hashing all of my features (I suppose this would be suited best for a separate question?)
PCA - could be viable, but if I'm understanding correctly the cardinality after one-hot encoding is too great and PCA will throw an error. In fairness, I have not tried this yet.
Binning doesn't seem feasible since I could have a value of 3.5 or three and 1/2. Each one would be considered a separate bin and thus not a solution to my problem.
Thanks to all that can share their insight/opinion.