Return to Answer

I was learning this topic too, and these are what I found:

• This type of encoding is called likelihood encoding, impact coding or target coding

• The idea is encoding your categorical variable with the use of target variable (continuous or categorical depending on the task). For example, if you have regression task, you can encode your categorical variable with the mean of the target. For every category, you calculate the corresponding mean of the target (among this category) and replace the value of a category with this mean.

• If you have classification task, you calculate the relative frequency of your target with respect to every category value.

• From a mathematical point of view, this encoding means a probability of your target, conditional on each category value.

• If you do it in a simple way, how I described above, you will probably get a biased estimation. That's why in Kaggle community they usually use 2 levels of cross-validation. Read this comment by raddar here. The corresponding notebook is here.

The quote:

It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;

Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.

So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.

It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)

• Another implementation of this encoding is here.

• In R library vtreat they have implementation of impact encoding. See this post.

• In CatBoost library they have a lot of options for categorical variable encoding including target encoding.

• There is no such encoding in sklearn yet.

I was learning this topic too, and these are what I found:

• This type of encoding is called likelihood encoding, impact coding or target coding

• The idea is encoding your categorical variable with the use of target variable (continuous or categorical depending on the task). For example, if you have regression task, you can encode your categorical variable with the mean of the target. For every category, you calculate the corresponding mean of the target (among this category) and replace the value of a category with this mean.

• If you have classification task, you calculate the relative frequency of your target with respect to every category value.

• From a mathematical point of view, this encoding means a probability of your target, conditional on each category value.

• If you do it in a simple way, how I described above, you will probably get a biased estimation. That's why in Kaggle community they usually use 2 levels of cross-validation. Read this comment by raddar here. The corresponding notebook is here.

The quote:

It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;

Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.

So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.

It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)

• Another implementation of this encoding is here.

• In R library vtreat they have implementation of impact encoding. See this post.

• In CatBoost library they have a lot of options for categorical variable encoding including target encoding.

• There is no such encoding in sklearn yet.

UPDATE: There is a nice package for sklearn models and pipelines! https://github.com/scikit-learn-contrib/category_encoders

I was learning this topic too, and these are what I found:

• This type of encoding is called likelihood encoding, impact coding or target coding

• The idea is encoding your categorical variable with the use of target variable (continuous or categorical depending on the task). For example, if you have regression task, you can encode your categorical variable with the mean of the target. For every category, you calculate the corresponding mean of the target (among this category) and replace the value of a category with this mean.

• If you have classification task, you calculate the relative frequency of your target with respect to every category value.

• From a mathematical point of view, this encoding means a probability of your target, conditional on each category value.

• If you do it in a simple way, how I described above, you will probably get a biased estimation. That's why in Kaggle community they usually use 2 levels of cross-validation. Read this comment by raddar here. The corresponding notebook is here.

The quote:

It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;

Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.

So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.

It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)

• Another implementation of this encoding is here.

• From a mathematical point of view, this target encoding means a probability of your target, conditional on each category value.

• In R library vtreat they have implementation of impact encoding. See this post.

• In CatBoost library they have a lot of options for categorical variable encoding including target encoding.

• There is no such encoding in sklearn yet.

I was learning this topic too, and these are what I found:

• This type of encoding is called likelihood encoding, impact coding or target coding

• The idea is encoding your categorical variable with the use of target variable (continuous or categorical depending on the task). For example, if you have regression task, you can encode your categorical variable with the mean of the target. For every category, you calculate the corresponding mean of the target (among this category) and replace the value of a category with this mean.

• If you have classification task, you calculate the relative frequency of your target with respect to every category value.

• From a mathematical point of view, this encoding means a probability of your target, conditional on each category value.

• If you do it in a simple way, how I described above, you will probably get a biased estimation. That's why in Kaggle community they usually use 2 levels of cross-validation. Read this comment by raddar here. The corresponding notebook is here.

The quote:

It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;

Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.

So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.

It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)

• Another implementation of this encoding is here.

• In R library vtreat they have implementation of impact encoding. See this post.

• In CatBoost library they have a lot of options for categorical variable encoding including target encoding.

• There is no such encoding in sklearn yet.

I was learning this topic too, and these are what I found:

• This type of encoding is called likelihood encoding, impact coding or target coding

• The idea is encoding your categorical variable with the use of target variable (continuous or categorical depending on the task). For example, if you have regression task, you can encode your categorical variable with the mean of the target. For every category, you calculate the corresponding mean of the target and replace the value of category with this mean.

• If you do it in a simple way, how I described above, you will probably get a biased estimation. That's why in Kaggle community they usually use 2 levels of cross-validation. Read this comment by raddar here. The corresponding notebook is here.

• Another implementation of this encoding is here.

• From a mathematical point of view, this target encoding means a probability of your target, conditional on each category value.

• In R library vtreat they have implementation of impact encoding. See this post.

• In CatBoost library they have a lot of options for categorical variable encoding including target encoding.

I was learning this topic too, and these are what I found:

• This type of encoding is called likelihood encoding, impact coding or target coding

• The idea is encoding your categorical variable with the use of target variable (continuous or categorical depending on the task). For example, if you have regression task, you can encode your categorical variable with the mean of the target. For every category, you calculate the corresponding mean of the target (among this category) and replace the value of a category with this mean.

• If you have classification task, you calculate the relative frequency of your target with respect to every category value.

• From a mathematical point of view, this encoding means a probability of your target, conditional on each category value.

• If you do it in a simple way, how I described above, you will probably get a biased estimation. That's why in Kaggle community they usually use 2 levels of cross-validation. Read this comment by raddar here. The corresponding notebook is here.

The quote:

It's taking mean value of y. But not plain mean, but in cross-validation within cross-validation way;

Let's say we have 20-fold cross validation. we need somehow to calculate mean value of the feature for #1 fold using information from #2-#20 folds only.

So, you take #2-#20 folds, create another cross validation set within it (i did 10-fold). calculate means for every leave-one-out fold (in the end you get 10 means). You average these 10 means and apply that vector for your primary #1 validation set. Repeat that for remaining 19 folds.

It is tough to explain, hard to understand and to master :) But if done correctly it can bring many benefits:)

• Another implementation of this encoding is here.

• From a mathematical point of view, this target encoding means a probability of your target, conditional on each category value.

• In R library vtreat they have implementation of impact encoding. See this post.

• In CatBoost library they have a lot of options for categorical variable encoding including target encoding.

• There is no such encoding in sklearn yet.