I am trying to understand how I can encode categorical variables using likelihood estimation, but have had little success so far.
Any suggestions would be greatly appreciated.
$\begingroup$
$\endgroup$
1
-
$\begingroup$ What to do while prediction time when we don't have target label? $\endgroup$– Ranjeet SinghCommented Mar 6, 2018 at 10:30
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
4
I was learning this topic too, and these are what I found:
This type of encoding is called likelihood encoding, impact coding or target coding
The idea is encoding your categorical variable with the use of target variable (continuous or categorical depending on the task). For example, if you have regression task, you can encode your categorical variable with the mean of the target. For every category, you calculate the corresponding mean of the target (among this category) and replace the value of a category with this mean.
If you have classification task, you calculate the relative frequency of your target with respect to every category value.
From a mathematical point of view, this encoding means a probability of your target, conditional on each category value.
If you do it in a simple way, how I described above, you will probably get a biased estimation. That's why in Kaggle community they usually use 2 levels of cross-validation. Read this comment by raddar here. The corresponding notebook is here.
The quote:
It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;
Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.
So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.
It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)
Another implementation of this encoding is here.
In R library vtreat they have implementation of impact encoding. See this post.
In CatBoost library they have a lot of options for categorical variable encoding including target encoding.
There is no such encoding in sklearn yet.
UPDATE: There is a nice package for sklearn models and pipelines! https://github.com/scikit-learn-contrib/category_encoders
-
1$\begingroup$ There is Target Encoding in Sklearn-contrib Category Encoders $\endgroup$– joshCommented Apr 13, 2018 at 10:28
-
$\begingroup$ How would you implement feature interaction in case you use target encoding? For example, you target-encoded F1 and F2. Would you just multiply the encoded values F1*F2? $\endgroup$ Commented Jul 17, 2018 at 19:45
-
$\begingroup$ If you calculate the mean for each LOO fold, then you take average of them, it is exactly the same as you are taking the mean of #2-#20 fold, I dont see why this can be considered as CV. Also I dont understand what he means by "vector" when he averages those 10 means. $\endgroup$– SiXUlmCommented Feb 20, 2019 at 10:36
-
1$\begingroup$ A late comment; the target encoding in Category Encoders is a simple mean encoding; it does not perform the folds-within-folds regularization described by raddar. $\endgroup$ Commented Mar 24, 2019 at 7:40
$\begingroup$
$\endgroup$
Target encoding is now available in sklearn through the category_encoders package.
Target Encoder
class category_encoders.target_encoder.TargetEncoder(verbose=0, cols=None, drop_invariant=False, return_df=True, impute_missing=True, handle_unknown='impute', min_samples_leaf=1, smoothing=1)
Target Encode for categorical features. Based on leave one out approach.
As noted by josh in the comment above.
$\begingroup$
$\endgroup$
Likelihood encoding is still not available on scikit learn. You may do it by creating a dictionary, and then do a replace function.
dict1 = {'cate1':1,'cate2':1,'cate3':2,'cate4':2}
for i in df.shape[0]:
df.iloc[i] = dict1[df.iloc[i]]
