I'm new to the datascience field and working on an assignment. I have a dataset with 150K rows with a categorical and numerical data, the target is a boolean. A categorical column consist of quite some codes which occur with random frequency (the most frequent appearing in 64K records, the lest in one record, quite a large number ) The number of categories having only 1 record in the dataset is quite large (35), and about 110 categories have 10 or less occurences;
>>> df.groupby('category')['target'].agg(['count','mean']).sort_values('count')
count mean
category
code__1 1 0.000000
//readabity
code_36 1 0.000000
code_37 2 0.000000
code_38 2 0.500000
//readability
code_103 10 0.000000
code_151 73 0.000000
code_175 706 0.247875
code_188 23145 0.032059
code_189 64414 0.074006
For the total dataset we have:
>>> df['target'].agg(['mean','count'])
mean 0.072536
count 159880.000000
I'm a bit concerned that my dataset will grow out of hand when I'm using all categories and convert them to 189 binary columns with pd.get_dummies
. I'm thinking of throwing out all categories with less than 10 occurences and replacing them for a dummy category. Now my 3 questions:
- is this wise? will I introduce data leakage or mess up future datasets?
- is there a way to do this in sklearn/pandas? I'm thinking of a python approach by storing code103..189 into a list
cats
, iterate over my category column and substitute all categories not in thecats
list todummy
, this is reproducible over a future set or test set. - Another approach would be to replace all categories to
dummie
where themean
is close to the total mean of target column? then I would ditch code_189 (the most frequent) because its varies barely from 0.072?