0
$\begingroup$

This is the problem of binary classification: "1" - the subscriber is a driver (belongs to the segment of drivers), "0" - the subscriber is not a driver (does not belong to the segment of drivers).

The files tabular_data.csv and hashed_feature.csv ̶ here are descriptive characteristics for 4084 subscribers ("ID" is the subscriber ID). The train.csv file is data about the target label (subscriber id and corresponding binary target).

The tabular_data.csv file contains numerical data on the subscriber's activity for 12 periods. • period - period number (consecutive periods, 1 - the newest); • id - subscriber ID; • feature_0 - feature_49 - data on the subscriber's activity in the corresponding period.

The hashed_feature.csv file is a set of hashed values of one categorical variable for the subscriber. • id - subscriber ID; • feature_50 - hash from the value of the categorical variable.

I faced the following problems:

  1. How should I encode feature_50 for ML algorithm? This feature contains approximately 5k unique values and each user can have almost 1k of these values. As I think one hot encoding is not very helpful here, but what technique should I use in this case?
  2. NA filling strategy for feature_0 - feature_49 (I've only tried pandas filling strategies)
  3. Make use of period variable? I would appreciate any help.
$\endgroup$

1 Answer 1

1
$\begingroup$

When you have high cardinality I suggest you two options (one doesn't exclude the other):

  • Aggregate least frequent categories into one called other
  • Use Neural Network with an embedding layer for high cardinality categories. In case you don't know what an embedding is, it is just a table that map each category into a vector of k features (k your choice)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.