0
$\begingroup$

How do I perform frequency/count encoding for a train and test set?

The implementations of this encoding I've seen simply frequency encode the categorical variables on a particular dataset (no specific train, and test encoding transformation). For instance:

dataset.groupby("cat_column").size()/len(dataset)

In my case now I have a train, and test set.

[First option] Is it okay (due to leakage? or there won't?) for me to use frequency encoding on the whole dataset. OR

[Second option] I should take into consideration train, and test set independence.

If the second option, how do I do this?

  1. Encode the train set, then use the encoding values of categories in the train set for the same categories in the test set. Categories not represented in the test set would be need to be handled. OR
  2. There's a better generic implementation?
$\endgroup$
1
$\begingroup$

I hope this is not a late answer but actually you can use category_encoders library, it follows sklearn's style.

Example:

import category_encoders as ce 

#I'll pretend that you've already split your data into train/test

#your categorical features
cat_features = ['cat_feature1', 'cat_feature2']

#count encoder 
count_encoder = ce.CountEncoder(cols=cat_features)
count_encoder.fit(train[cat_features])

train = train.join(count_encoder.transform(train[cols]).add_suffix('_count'))
test = test.join(count_encoder.transform(test[cols]).add_suffix('_count'))

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.