1
$\begingroup$

I'm new to the datascience field and working on an assignment. I have a dataset with 150K rows with a categorical and numerical data, the target is a boolean. A categorical column consist of quite some codes which occur with random frequency (the most frequent appearing in 64K records, the lest in one record, quite a large number ) The number of categories having only 1 record in the dataset is quite large (35), and about 110 categories have 10 or less occurences;

>>> df.groupby('category')['target'].agg(['count','mean']).sort_values('count')
                      count      mean
category                       
code__1                   1  0.000000
//readabity
code_36                   1  0.000000
code_37                   2  0.000000
code_38                   2  0.500000
//readability
code_103                 10  0.000000
code_151                 73  0.000000
code_175                706  0.247875
code_188              23145  0.032059
code_189              64414  0.074006

For the total dataset we have:

>>> df['target'].agg(['mean','count'])
mean          0.072536
count    159880.000000

I'm a bit concerned that my dataset will grow out of hand when I'm using all categories and convert them to 189 binary columns with pd.get_dummies. I'm thinking of throwing out all categories with less than 10 occurences and replacing them for a dummy category. Now my 3 questions:

  1. is this wise? will I introduce data leakage or mess up future datasets?
  2. is there a way to do this in sklearn/pandas? I'm thinking of a python approach by storing code103..189 into a list cats, iterate over my category column and substitute all categories not in the cats list to dummy, this is reproducible over a future set or test set.
  3. Another approach would be to replace all categories to dummie where the mean is close to the total mean of target column? then I would ditch code_189 (the most frequent) because its varies barely from 0.072?
$\endgroup$

1 Answer 1

1
$\begingroup$

The most straightforward approach would be to consider these small categories as single 'unknown' category - if they're really small, your model shouldn't change drastically.

Another approach would be to replace all categories to dummie where the mean is close to the total mean of target column? then I would ditch code_189 (the most frequent) because its varies barely from 0.072?

This idea seems to be close to mean encoding approach. Mean encoding consists of creating a column where categories are encoded by mean of target for this category. This can incorporate knowledge of small categories naturally - you can just replace their means with global mean, or use smoothing. I encourage you to see this segment on mean encodings (the next video is on regularization).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.