1
$\begingroup$

I know this question has been asked before and I have tried a few things but those things are not working as expected for my usecase.

I have a 500 length feature vector. One of these features is a categorical value pincode. For our dataset, the pincode can take more than 20,000 unique values. So we can't used one hot encoding because it will blow up our feature space.

I have also tried Binary Encoding which assigns a unique integer to every unique categorical value and then converts it into Binary. It then treats every bit as a column. This way the dimension of the pincode feature vector had reduced from 20,000 to around 20. But I didn't find any improvement in our evaluation metrics which is AUC (Area under Curve). Intuitively also it didn't felt right thing to do.

I also tried to apply PCA on the pincode to reduce the dimensions. I tried to reduce the dimensions from 20,000 to 100. Upon applying PCA, the model performed worse. Also, I read somewhere that PCA doesn't work well on categorical values. It's better for continous values.

So what do we do with this feature? We don't want to throw it away as we think that it might be an important feature. But we want to reduce it dimension and then use.

Apologies if this is a basis question. I am new to this field and trying various things out.

P.S - We are using xgboost for training our model.

$\endgroup$
4
  • $\begingroup$ What kind of algorithm are you using? $\endgroup$
    – Dan Scally
    Commented Sep 20, 2019 at 9:11
  • $\begingroup$ we are using xgboost for training. $\endgroup$ Commented Sep 20, 2019 at 9:13
  • $\begingroup$ Is pincode a feature that is used to identify objects? Like a user ID? $\endgroup$
    – m13op22
    Commented Feb 17, 2020 at 17:26
  • $\begingroup$ Are the pincodes "adjacent" to one another in any meaningful way? Like does 2095678 refer to something that sits next to or is similar to 2095679 or whatever? If so, something like locality sensitive hashing might be an option. Mush your pincodes into buckets and then treat the buckets as your categories. $\endgroup$
    – Jeremiah
    Commented Dec 20, 2023 at 20:04

4 Answers 4

1
$\begingroup$

one idea is to craft some features of this feature - when you classify cars you don't have data like strings "Ferrari 991 year 2014 red" "BMW z4 year 1999 2.0L blue" but you would like to have columns like "Manufacturer", "Type(SUV/Cabrio...)", "Year", "Engine" etc.

you could transform it or just discard

$\endgroup$
1
$\begingroup$

Take a look at this research paper.

It mentions two methods, a Minhash Encoding technique and Gamma-Poisson Matrix Factorization technique for high cardinality categorical data.

$\endgroup$
1
$\begingroup$

Pincode I feel very very likely is not a predictor at all for the target variable you have. If you feel geography is an important predictor and has bearing on you target variable then what you can do is use city or state or other relevant geographical unit like sub-city etc. you feel have homogeneous characterstics.

Say if state X n Y each has 2 cities P,Q and M,N and ecah city is very different from other one then I will choose city as my geographical variable otherwise state as Geo variable is fine. So it depends on homogeneity of Geo unit.

For example if say you are predicting demand for luxury cars and city A has great roads, high income population etc. and thus high demand for luxury cars compared to city B which has none of features city A has. So in your model city A observations have more probability of buying a luxury car then city B observations.

So by using a bigger Geo unit than pincode you end up reducing categories say from 20000 pin codes you can have only 100 cities which is easily managable by xgb.

So use a bigger Geo unit if you feel geography affects your target variable.

$\endgroup$
1
  • $\begingroup$ Note that in some countries, pincodes are not attributed randomly and can thus be good geographical predictors, once handled correctly. $\endgroup$ Commented Mar 26, 2020 at 9:22
0
$\begingroup$

So for dealing with lot of categorical features you can try some techniques:

  • Select K best features
  • PCA
  • Features importance
  • or different types of encoding methods

Let me give you one example, as i was watching one interview on data science, and the interviewer asked candidate,

*You are having pincode as category variable as that column contains over 500 unique values, so how will you handle it as you can't convert that using one-hot encoding,

My thought was to use: Target encoding method

please correct me if i am wrong.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.