Dropping less frequently used categorical data?

Question

I'm new to the datascience field and working on an assignment. I have a dataset with 150K rows with a categorical and numerical data, the target is a boolean. A categorical column consist of quite some codes which occur with random frequency (the most frequent appearing in 64K records, the lest in one record, quite a large number ) The number of categories having only 1 record in the dataset is quite large (35), and about 110 categories have 10 or less occurences;

>>> df.groupby('category')['target'].agg(['count','mean']).sort_values('count')
                      count      mean
category                       
code__1                   1  0.000000
//readabity
code_36                   1  0.000000
code_37                   2  0.000000
code_38                   2  0.500000
//readability
code_103                 10  0.000000
code_151                 73  0.000000
code_175                706  0.247875
code_188              23145  0.032059
code_189              64414  0.074006

For the total dataset we have:

>>> df['target'].agg(['mean','count'])
mean          0.072536
count    159880.000000

I'm a bit concerned that my dataset will grow out of hand when I'm using all categories and convert them to 189 binary columns with pd.get_dummies. I'm thinking of throwing out all categories with less than 10 occurences and replacing them for a dummy category. Now my 3 questions:

1. is this wise? will I introduce data leakage or mess up future datasets?
2. is there a way to do this in sklearn/pandas? I'm thinking of a python approach by storing code103..189 into a list cats, iterate over my category column and substitute all categories not in the cats list to dummy, this is reproducible over a future set or test set.
3. Another approach would be to replace all categories to dummie where the mean is close to the total mean of target column? then I would ditch code_189 (the most frequent) because its varies barely from 0.072?

Answer 1

The most straightforward approach would be to consider these small categories as single 'unknown' category - if they're really small, your model shouldn't change drastically.

Another approach would be to replace all categories to dummie where the mean is close to the total mean of target column? then I would ditch code_189 (the most frequent) because its varies barely from 0.072?

This idea seems to be close to mean encoding approach. Mean encoding consists of creating a column where categories are encoded by mean of target for this category. This can incorporate knowledge of small categories naturally - you can just replace their means with global mean, or use smoothing. I encourage you to see this segment on mean encodings (the next video is on regularization).