Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [categorical-encoding]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

User Behaviour Anomaly Detection

I´m trying to detect anomalies in the behaviour of users in an app. My dataset have several fields, but I think the most important ones are User Id, TimeStamp, and Event_name. So for example I could ...
Juan Pablo Pereira's user avatar
0 votes
2 answers
54 views

Encode 10k features where each feature is having more than 500 categories

I have around 10k features in my dataset and each feature is having more than 500 categories. what is the best encoding method to convert this categorical features to vector form? "span_dir":...
khushi's user avatar
  • 111
0 votes
0 answers
8 views

Best way to encode a tag column for clustering

I have a dataset which tells me a tech support case used a particular tech document. Every case has been tagged with which product it pertains to. Similarly tech documents are tagged with certain key ...
haldar55's user avatar
4 votes
1 answer
823 views

Decision Tree only splits to the left

I can’t really understand, why my decision tree only splits to the left. I originally have 2 categorical features (further named feature 0 and 1), which I concat to one feature since feature 1 is ...
Taitex's user avatar
  • 41
0 votes
0 answers
25 views

Binary classification using xgboost

Why when adding new features in my ADS for a binary classification using XGBOOST my score and uplift has decreased ? What is the best way to treate categorical features or other features in order that ...
Warda_IDRIS's user avatar
0 votes
2 answers
29 views

How to encode Income Type Ordinal Data into numbers?

I am doing a mini project on Credit card Approval Prediction. The Dataset I have taken is from Kaggle: https://www.kaggle.com/datasets/rikdifos/credit-card-approval-prediction Problem: I want to ...
Prajwal Dhage's user avatar
0 votes
0 answers
28 views

Encode multiple label categorical variables with consideration of the frequency and standardization

I'm currently working on a dataset containing, among other features, multi-labeled categorical data per person for the last 3 years. I'm not sure how to handle this kind of categorical data where ...
codade's user avatar
  • 1
1 vote
2 answers
1k views

Can decision trees handle Nominal Categorical variables?

I have read that decision trees can handle categorical columns without encoding them. However, as decision trees make splits on the data, how does it handle Nominal Categorical variables? Surely a ...
Connor's user avatar
  • 651
0 votes
0 answers
27 views

Multi-class classification model unable to return desired outcome

I have a scenario of multi-class dataset with around 10 distinct classes of target. There are 3 categorical features each with multiple labels. If we check the data, each unique combination of feature ...
soumalya saha's user avatar
1 vote
1 answer
32 views

How to prevent my model from mistaking categorical feature for ordinal feature

I have tabular data where each group of 100 rows represents a deployment of a specific geometry that has certain features measured. For example, I have 10000 deployments stored in a column called &...
Hamdi Barkous's user avatar
1 vote
1 answer
183 views

Revisit - Encoding before vs after train test split?

Background: Like a good data scientist, I planned to fit my encoder on my training set and use it to transform my test set. However, when I tried to transform my test set, the encoder threw an error....
Snehal Patel's user avatar
0 votes
0 answers
14 views

How to encode data set with all categorical predictors

I have a data set with all categorical predictors. They are 14 in number. If I do one-hot encoding, I would be getting more than 35 new features, which I think is not right due to the curse of ...
emekadavid's user avatar
0 votes
0 answers
554 views

Binary classification works with softmax, but not sigmoid

I am doing a binary classification problem for seizure classification. I split the data into Training, Validation and Test with the following sizes and shapes dataset_X = ...
Mohammed Nafie's user avatar
1 vote
1 answer
74 views

N-ary decision tree with categorical features

I want to build an n-ary decision tree with categorical features. I am using ordinary ID3 algorithm to build a tree. Lets take the next dataset as a training dataset for building a decision tree: ...
dzi's user avatar
  • 111
0 votes
1 answer
28 views

How do I use one hot encoding with 240 nominal variables and each with equal occurrence?

The method I saw that's generally used to deal with large # of nominal variables is to keep the most frequent variables and introduce a new "other" category. But that's not possible with my ...
learner's user avatar
2 votes
1 answer
432 views

Encoding for Linear Regression

I have a CSV file with salary information and other columns. I am trying to transform some of these columns into proper values, for a ...
Alix Blaine's user avatar
1 vote
1 answer
144 views

If I use Weight of Evidence to transform categorical variables, do I still need to inform their indexes to Catboost

I'm using Weight of Evidence (WOE) to encode my categorical features. Do I still need to inform Catboost that they are categorical features by using cat_features parameter?
Jorge Amaral's user avatar
1 vote
0 answers
50 views

Intuition behind catboost encoding techniques

Can anyone please help me in understanding the effect of various bucketing techniques used in CatBoost Algorithm for categorical features? Like there is border, buckets, binarized target mean, counter ...
Mimansa Maheshwari's user avatar
0 votes
0 answers
68 views

Encode and replace missing categorical values

Should replacing missing values for categorical variables be done after one-hot-encoding? (ex: I have a column named Car Type: Sedan, Hatchback, MPV, SUV) which has 5% missing values. Would there be ...
Kusisi Karem's user avatar
0 votes
1 answer
126 views

how to handle categorical data that has two or more columns with unique values?

...
Deep Kumar Prasad's user avatar
0 votes
1 answer
931 views

Turning multiple binary columns into categorical (with less columns) with Python Pandas

I want to turn these categories into values of categorical columns. The values in each category are the current binary columns present in the data frame. We have : A11, A12.. is a detail of A1 so if ...
Legna's user avatar
  • 21
0 votes
1 answer
40 views

Dealing with observation with arbitrary number of categories with arbitary number of values

Suppose to have a set of elements $X = \{x_1, x_2, ..., x_n\}$. Each element is characterised by a set of features. The features characterising a particular element $x_i$ can belong to one of $q$ ...
King Powa's user avatar
  • 111
2 votes
0 answers
272 views

Does it make sense to use target encoding together with tree-based models?

I'm working on a regression problem with a few high-cardinality categorical features (Forecasting different items with a single model). Someone suggested to use target-encoding (mean/median of the ...
KJA's user avatar
  • 21
1 vote
0 answers
238 views

Encode each comma separated value in Pandas

I have a dataset ...
spd's user avatar
  • 119
-1 votes
1 answer
557 views

How to deal with address (like zip-code) for training a model?

To me it doesn't make sense to normalize it even if it is a numerical variable like Zip Code. An address should be interpreted as categorical features like "neighborhood"... ? Suppose I have ...
aRedDish's user avatar
2 votes
1 answer
35 views

How do I get the mean values that are greater than .5 for my model?

I am trying to build a classification model. One of the variables called specialty has 200 values. Based on a previous post I saw, I decided I wanted to include the values that have the highest mean. ...
bulldog23's user avatar
1 vote
0 answers
154 views

Handling date and time fields for classification task

I'm working on a classification task(The dataset is 400,000 rows and 30 columns) and one of my features was date-time. I've extracted the month, day of the week, and hour from the dataset (year is a ...
insomniac's user avatar
5 votes
2 answers
8k views

How to handle categorical variables with Random Forest using Scikit Learn?

One of the variables/features is the department id, which is like 1001, 1002, ..., 1218, etc. The ids are nominal, not ordinal, i.e., they are just ids, department 1002 is by no means higher than ...
Fred Chang's user avatar
0 votes
1 answer
233 views

NAN in keras neural network results

I am creating a neural network simple architecture. But I keep getting NAN in result, cant figure out why, below is my code. ...
Ayan Mitra's user avatar
2 votes
2 answers
296 views

Categorical Variable Embedding

I have a categorical variable in my labeled dataset. I trained one-hot encoded version of it in another neural network having embedding layer with a larger labeled dataset. I have obtained the weights ...
Agile's user avatar
  • 21
0 votes
1 answer
225 views

Collapsing categorical data into more than 3 categories

I have a bunch of categorical, part of speech data that I want to collapse into fewer categories. np.where() won't do because I want to have 6 categories at the end: noun, verb, adjective, adverb, ...
macarom's user avatar
0 votes
1 answer
108 views

What's the minimum percentage of categories should be present in the categorical variable for to ignore the variable entirely

For example, if i have a feature "colour_codes" that has close to 5000 distinct color codes inside it. And the number of samples/rows is 10 million. Then should I ignore the feature "...
insomniac's user avatar
1 vote
1 answer
272 views

Categorical feature encoding

I am making a classification model. I have categorical and continuous data. The categorical columns include columns with 2 classes such as sex (male, female), and multi-class columns such as location. ...
Alis's user avatar
  • 11
0 votes
1 answer
185 views

Separating numerical and categorical features in a binary classification problem

I have a dataset with employee data with around 9500 rows, and have to predict if the target is 0 or 1. Some of my features are the department of an employee, gender, salary, review_score(numerical),...
Bluetail's user avatar
  • 101
1 vote
2 answers
427 views

Encoding features with big amount of classes

Is it worth to encode features with big amount of classes ( such as 60 )? Or should I leave it as it is ?
A Arbitrage's user avatar
0 votes
3 answers
4k views

Dealing with Extra Categories in Test Set

Suppose I have a data set which consists a dependent variable y and independent variables X. Suppose that there is a specific ...
Sam OT's user avatar
  • 109
0 votes
1 answer
477 views

What are the options/best practices for encoding categorical features for multilabel classification?

I am working on a multilabel classification problem with both continuous and categorical features. For a single label problem, I might make use of a supervised encoder for my categorical features such ...
tensormoby's user avatar
2 votes
2 answers
186 views

Decision Trees and Categorical Feature Labelling

I am working on a decision tree model and trying to decide how best to handle categorical features. The features in my dataset are generally high in cardinality and I have found that ordinal labeling ...
dbdata's user avatar
  • 21
2 votes
1 answer
372 views

Two questions about one-hot encoding: drop first? and features with thousands of categories [closed]

I have two questions about one-hot feature encoding: (1) Is it considered a "best practice" to drop the first (or at least one) one-hot encoded feature when one-hot encoding, like you would ...
Andrew Bell's user avatar
0 votes
1 answer
268 views

What is the best practice to normalize/standardize imbalanced data for outlier detection or binary classification task?

I'm researching Anomaly/outlier/fraud detection, and I'm looking for the best practice to pre-process the synthetic data for imbalanced data. I have checked all methodology for normalizing/...
Mario's user avatar
  • 400
0 votes
1 answer
478 views

Handling categorical data with more over 100 unique classes

I am working with a pure categorical data set. And some classes have more than 100 unique values. I could not find any appropriate encoding possibility. So I created a SQL table, where each value got ...
Julia's user avatar
  • 3
1 vote
1 answer
143 views

Categorical encoded variables in scikit-learn diabetes dataset [closed]

When using sklearn.datasets import load_diabetes, the sex variable which is categorical, is scaled to continuous values. Is it even legal to scale such variables?
Ashish Rai's user avatar
2 votes
2 answers
747 views

Is it acceptable to use label encoding for nominal categorical data when one hot encoding would create too many features?

I'm working on a short data science project to compare the accuracy of different classification methods. The groups decided to use and compare Random Forest, Naive Bayes and SVM. The dataset we are ...
Jim Jones's user avatar
1 vote
2 answers
1k views

Different number of features in train vs test when using Label Encoding

This is not a duplicate of Different number of features in train vs test There are some categorical columns in my data, and the cardinality for each of them is large, so I chose to use LabelEncoding ...
hamnghi's user avatar
  • 13
2 votes
1 answer
606 views

kNN for non-ordinal variables

kNN is a distance-based method, so it requires the input to be in numerical form. I was wondering if it is possible to use kNN imputer for non-ordinal categorical variables (like color). Since the ...
Szymon Adamik's user avatar
2 votes
1 answer
324 views

Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less ...
Vladislav Gladkikh's user avatar
0 votes
0 answers
654 views

Tensorflow - I don't get the right shapes - `ValueError: Shapes (9, 1) and (8, 9) are incompatible`

I want to train a Sequential Neural Net (NN) with Tensorflow. ...
Dominik's user avatar
3 votes
1 answer
65 views

One hot encoding of target variable containing classes 1 to 9 not including zero

While predicting a solution for a sudoku puzzle using CNN, the target variable should predict values from 1 to 9 for all the 81(9*9) values in the puzzle. Hence the target value shape is (81,9). Using ...
Sathish Kumar SG's user avatar
8 votes
5 answers
7k views

How do I encode the categorical columns if there are more than 15 unique values?

I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have ...
Cinemato's user avatar
0 votes
1 answer
509 views

why should i do target encoding within cv loop?

i wish to use target encoding, using the category encoders sklearn library. I don't really understand why it is necessary to include this as a step in a sklearn pipeline WITHIN the cross validation ...
Maths12's user avatar
  • 526