Skip to main content
Data Science

Return to Question

Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
Bumped by Community user
added 375 characters in body
Source Link
Nick Bohl
Nick Bohl
  • 95
  • 4
   Product Height                 Product Description
  ---------               ------------------------
   Ball three and a half inches            Round bouncy toy for kids
   Bat three and 1               Stick that kids use to hit a ball
   Go-Kart red/2black inches   Small motorized vehicle for kids
   Go-Kart 27.6234blue/green inches  Small motorized vehicle for kids
   Wrench tall              Tool for tightening or loosening bolts
   Ratchet  short            Tool for tightening or loosening bolts
   Reclining kindaarm-chair short Cushioned seat for lounging

Unfortunately, I cannot confirm or disprovethink that the descriptions are standardized if they fall within a particular category but at this time I cannot confirm if there are a finitethe number of possibleunique descriptions for any of these description valuesare finite. At this time, my assumption would be to treat these as nominal-categorical, as these are literally descriptive and not qualitative.

    Height Description
--------------------------
    three and a half inches
    three and 1/2 inches
    27.6234 inches
    tall
    short
    kinda short

Unfortunately, I cannot confirm or disprove at this time if there are a finite number of possible descriptions for any of these description values. At this time, my assumption would be to treat these as nominal-categorical, as these are literally descriptive and not qualitative.

   Product                  Product Description
  ---------               ------------------------
   Ball                 Round bouncy toy for kids
   Bat                  Stick that kids use to hit a ball
   Go-Kart red/black    Small motorized vehicle for kids
   Go-Kart blue/green   Small motorized vehicle for kids
   Wrench               Tool for tightening or loosening bolts
   Ratchet              Tool for tightening or loosening bolts
   Reclining arm-chair  Cushioned seat for lounging

I think that the descriptions are standardized if they fall within a particular category but at this time I cannot confirm if the number of unique descriptions are finite. At this time, my assumption would be to treat these as nominal-categorical, as these are literally descriptive and not qualitative.

Source Link
Nick Bohl
Nick Bohl
  • 95
  • 4

How to leverage description data in multi-class classification (dimensionality reduction)

I'm currently working with a dataset of 55k records and seven columns (one target variable), three of which are nominal categorical. The other three are 'description' fields with high cardinality, as would be expected in many cases of description data:

in>>
df[['size description', 'weight Description', 'height Description']].nunique()

out>>
 size Description       4066
 weight Description      736
 height Description     3173
 dtype: int64

some examples of these values could be:

    Height Description
--------------------------
    three and a half inches
    three and 1/2 inches
    27.6234 inches
    tall
    short
    kinda short

Unfortunately, I cannot confirm or disprove at this time if there are a finite number of possible descriptions for any of these description values. At this time, my assumption would be to treat these as nominal-categorical, as these are literally descriptive and not qualitative.

To that end, my question is what are some best practices for handling categorical features such as these?

Things I have considered:

  1. Label encoding is obviously not viable in this situation, as the descriptions have no hierarchy.

  2. One-hot encoding seems an unlikely solution as it balloons the shape of the dataset from (55300 , 6) to (55300 , 65223) due to the high cardinality of the other variables. However, I tried it anyway and generated 98% accuracy on my test set but very poor results on an out-of-sample validation set (5k records, 0-5% accuracy). Seems pretty clear it's over-fitting and thus, not viable.

  3. Hashing, for whatever reason, will not apply to one of the columns, but I suppose it could still be viable. I just need to figure out why it's not hashing all of my features (I suppose this would be suited best for a separate question?)

  4. PCA - could be viable, but if I'm understanding correctly the cardinality after one-hot encoding is too great and PCA will throw an error. In fairness, I have not tried this yet.

  5. Binning doesn't seem feasible since I could have a value of 3.5 or three and 1/2. Each one would be considered a separate bin and thus not a solution to my problem.

Thanks to all that can share their insight/opinion.