I'm new to the datascience field and working on an assignment. I have a dataset with 150K rows with a categorical and numerical data, the target is a boolean. A categorical column consist of quite some codes which occur with random frequency (the most frequent appearing in 64K records, the lest in one record, quite a large number ) The number of categories having only 1 record in the dataset is quite large (35), and about 110 categories have 10 or less occurences;
>>> df.groupby('category')['target'].agg(['count','mean']).sort_values('count') count mean category code__1 1 0.000000 //readabity code_36 1 0.000000 code_37 2 0.000000 code_38 2 0.500000 //readability code_103 10 0.000000 code_151 73 0.000000 code_175 706 0.247875 code_188 23145 0.032059 code_189 64414 0.074006
For the total dataset we have:
>>> df['target'].agg(['mean','count']) mean 0.072536 count 159880.000000
I'm a bit concerned that my dataset will grow out of hand when I'm using all categories and convert them to 189 binary columns with
pd.get_dummies. I'm thinking of throwing out all categories with less than 10 occurences and replacing them for a dummy category. Now my 3 questions:
- is this wise? will I introduce data leakage or mess up future datasets?
- is there a way to do this in sklearn/pandas? I'm thinking of a python approach by storing code103..189 into a list
cats, iterate over my category column and substitute all categories not in the
dummy, this is reproducible over a future set or test set.
- Another approach would be to replace all categories to
meanis close to the total mean of target column? then I would ditch code_189 (the most frequent) because its varies barely from 0.072?