8
$\begingroup$

I am working with a dataset with large number of categorical features (>80%) predicting a continuous target variable (i.e. Regression). I have been reading quite a bit about ways to handle categorical features. And learned that one-hot encoding I have been using in past is really bad idea especially when it comes to lots of categorical features with many levels (read these post, and this).

While I've come across methods like target-based encoding (smoothing) of categorical features often based on mean of target values for each feature perhaps this post/kernel in Kaggle. Still I am struggling to find a more concrete way till I found CatBoost an open-source gradient boosting on decision trees released last year by Yandex group. They seem to offer extra statistical counting options for categorical features likely much more efficient than simple one-hot encoding or smoothing.

The problem is the documentation is not helpful how to set CTR settings. I have tried different ways but it just does not work. The doc says CTR setting as simple_ctr, to be given as (CTR setting section):

['CtrType[:TargetBorderCount=BorderCount][:TargetBorderType=BorderType][:CtrBorderCount=Count][:CtrBorderType=Type][:Prior=num_1/denum_1]..[:Prior=num_N/denum_N]',
 'CtrType[:TargetBorderCount=BorderCount][:TargetBorderType=BorderType][:CtrBorderCount=Count][:CtrBorderType=Type][:Prior=num_1/denum_1]..[:Prior=num_N/denum_N]',
  ...]

Here is a super simple example, the data looks like this:

import pandas as pd
import catboost
data = [{'profit': '342','country': 'holland','account': 'Jones LLC', 'saving': 150, 'debt': -60, 'age': 28},
         {'profit': '875','country': 'germany','account': 'Alpha Co',  'saving': 200, 'debt': -10, 'age': 42},
         {'profit': '127','country': 'italy','account': 'Blue Inc',  'saving': 50,  'debt': -300,  'age': 38 }]
df = pd.DataFrame(data)

Here is a simple Catboost Regressor:

X_train = df.drop(['profit'],axis=1)
Y_train = df['profit']
categorical_features_indices = [0,2]

train_pool = catboost.Pool(X_train, Y_train, cat_features=categorical_features_indices)

model = catboost.CatBoostRegressor(
    depth=3,
    iterations=5,
    eval_metric='RMSE',
    simple_ctr=None)

model.fit(train_pool);

The simple_ctr, one of CTR settings, is the problem!! It is pity because it looks like it the package offers various methods, so far no way to access them.

UPDATE Aug. 9th, 2018: A few days ago I raised this problem to Catboost developer, see here, and they opened a ticket for it to provide a tutorial.

$\endgroup$
  • $\begingroup$ Here is the description about catboost on categorical vaiable: tech.yandex.com/catboost/doc/dg/concepts/… $\endgroup$ – DiveIntoML May 28 '18 at 0:07
  • $\begingroup$ Yes, I know the link; it gives a detailed theoretical description. Yet not enough description on how to use it is given; I mean practical usage instructions. I have have tried it a few months ago and it was not straightforward. $\endgroup$ – TwinPenguins May 28 '18 at 5:39
  • 1
    $\begingroup$ I am currently only using the one_hot_max_size param and my understanding is that everything that is not one hot encoded, will be encoded using target mean expanding encoding. I am looking forward to the additional official documentation to understand the ctr settings better and how I can play with them. $\endgroup$ – Garima Jain Mar 13 at 16:13
0
$\begingroup$

Did you try using the format provided like below:

['CtrType[:TargetBorderCount=BorderCount][:TargetBorderType=BorderType][:CtrBorderCount=Count][:CtrBorderType=Type][:Prior=num_1/denum_1]..[:Prior=num_N/denum_N]'

['BinarizedTargetMeanValue[:TargetBorderCount=1][:TargetBorderType=Uniform][:CtrBorderCount=5][:CtrBorderType=Uniform][:Prior=1]']
$\endgroup$
  • $\begingroup$ Is this an answer? It feels more like a question/comment. $\endgroup$ – Stephen Rauch Aug 9 '18 at 20:34
  • $\begingroup$ @Interested_Programmer: It does not work, try it yourself in the example I just provided. Have you tested it or you simply found it in the documentation? $\endgroup$ – TwinPenguins Aug 9 '18 at 21:13
  • $\begingroup$ I apologize @StephenRauch. I was unable to run my notebook but found it usable in other functions to follow the format. In hindsight, I should have added it as a comment. I am also waiting for the creators to put out the new tutorial. $\endgroup$ – Interested_Programmer Aug 10 '18 at 21:06
  • 1
    $\begingroup$ also this works. simple_ctr=['BinarizedTargetMeanValue']) $\endgroup$ – Interested_Programmer Aug 10 '18 at 21:20
  • 1
    $\begingroup$ Thanks. But as you shown it only works as simple_ctr=['BinarizedTargetMeanValue'])! The other options can not be fed like [:TargetBorderCount=BorderCount] etc. If you know how to feed others together with CtrType, let me know. $\endgroup$ – TwinPenguins Aug 11 '18 at 7:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.