Questions tagged [categorical-encoding]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
12 views

Does it make sense to use target encoding together with tree-based models?

I'm working on a regression problem with a few high-cardinality categorical features (Forecasting different items with a single model). Someone suggested to use target-encoding (mean/median of the ...
user avatar
  • 11
1 vote
0 answers
23 views

Encode each comma separated value in Pandas

I have a dataset ...
user avatar
  • 19
-1 votes
1 answer
29 views

How to deal with address (like zip-code) for training a model?

To me it doesn't make sense to normalize it even if it is a numerical variable like Zip Code. An address should be interpreted as categorical features like "neighborhood"... ? Suppose I have ...
user avatar
0 votes
0 answers
9 views

How to create the categorical mask for images specifically for Tensor? Or port the NumPy function correctly to Dataset.map function

I'm trying to move from NumPy array as my dataset to tensorflow.Dataset. Now, I've created a pipeline to train the model for classification problems. At some point, I just normalize all the images ...
user avatar
2 votes
1 answer
28 views

How do I get the mean values that are greater than .5 for my model?

I am trying to build a classification model. One of the variables called specialty has 200 values. Based on a previous post I saw, I decided I wanted to include the values that have the highest mean. ...
user avatar
1 vote
0 answers
13 views

Handling date and time fields for classification task

I'm working on a classification task(The dataset is 400,000 rows and 30 columns) and one of my features was date-time. I've extracted the month, day of the week, and hour from the dataset (year is a ...
user avatar
3 votes
2 answers
311 views

How to handle categorical variables with Random Forest using Scikit Learn?

One of the variables/features is the department id, which is like 1001, 1002, ..., 1218, etc. The ids are nominal, not ordinal, i.e., they are just ids, department 1002 is by no means higher than ...
user avatar
0 votes
0 answers
19 views

NAN in keras neural network results

I am creating a neural network simple architecture. But I keep getting NAN in result, cant figure out why, below is my code. ...
user avatar
1 vote
1 answer
16 views

Categorical Variable Embedding

I have a categorical variable in my labeled dataset. I trained one-hot encoded version of it in another neural network having embedding layer with a larger labeled dataset. I have obtained the weights ...
user avatar
  • 11
0 votes
1 answer
24 views

Collapsing categorical data into more than 3 categories

I have a bunch of categorical, part of speech data that I want to collapse into fewer categories. np.where() won't do because I want to have 6 categories at the end: noun, verb, adjective, adverb, ...
user avatar
0 votes
0 answers
41 views

How to group multiple categories of a categorical variable before feeding the data to a machine learning algorithm?

I have a labelled dataset to which I wish to fit a classification model (say, a Decision Tree). One of the categorical variables (say STATE) in the data has a lot ...
user avatar
0 votes
1 answer
22 views

What's the minimum percentage of categories should be present in the categorical variable for to ignore the variable entirely

For example, if i have a feature "colour_codes" that has close to 5000 distinct color codes inside it. And the number of samples/rows is 10 million. Then should I ignore the feature "...
user avatar
1 vote
1 answer
53 views

Categorical feature encoding

I am making a classification model. I have categorical and continuous data. The categorical columns include columns with 2 classes such as sex (male, female), and multi-class columns such as location. ...
user avatar
  • 11
0 votes
1 answer
34 views

Separating numerical and categorical features in a binary classification problem

I have a dataset with employee data with around 9500 rows, and have to predict if the target is 0 or 1. Some of my features are the department of an employee, gender, salary, review_score(numerical),...
user avatar
  • 101
0 votes
2 answers
25 views

Encoding features with big amount of classes

Is it worth to encode features with big amount of classes ( such as 60 )? Or should I leave it as it is ?
user avatar
0 votes
0 answers
38 views

How to deal with multiple binary timeseries?

I have a time series data looks like this userID month year target user1 1 2 1 user2 12 2 0 ... ... ... ... userN 6 3 0 with about 2000 unique userID, ...
user avatar
-1 votes
3 answers
172 views

Dealing with Extra Categories in Test Set

Suppose I have a data set which consists a dependent variable y and independent variables X. Suppose that there is a specific ...
user avatar
  • 99
0 votes
1 answer
48 views

What are the options/best practices for encoding categorical features for multilabel classification?

I am working on a multilabel classification problem with both continuous and categorical features. For a single label problem, I might make use of a supervised encoder for my categorical features such ...
user avatar
2 votes
2 answers
73 views

Decision Trees and Categorical Feature Labelling

I am working on a decision tree model and trying to decide how best to handle categorical features. The features in my dataset are generally high in cardinality and I have found that ordinal labeling ...
user avatar
  • 21
0 votes
0 answers
32 views

What is the recommended embedding for a categorical variable with more than 40000 thousand categories?

I have a feature called Planning_id with more than 40000 categories. What is the recommended embedding size? I read that: embedding_dimension = # categories * 0.25 is a good rule of thumb, but I still ...
user avatar
0 votes
0 answers
17 views

How to treat a column that contains a list of categorical, high cardinality values for a classification problem?

The list cannot be exploded into several columns because this will result in very high dimensionality. I would like to know the following: How to treat this kind of column in a dataframe? Can I keep ...
user avatar
  • 1
2 votes
1 answer
93 views

Two questions about one-hot encoding: drop first? and features with thousands of categories [closed]

I have two questions about one-hot feature encoding: (1) Is it considered a "best practice" to drop the first (or at least one) one-hot encoded feature when one-hot encoding, like you would ...
user avatar
0 votes
1 answer
78 views

What is the best practice to normalize/standardize imbalanced data for outlier detection or binary classification task?

I'm researching Anomaly/outlier/fraud detection, and I'm looking for the best practice to pre-process the synthetic data for imbalanced data. I have checked all methodology for normalizing/...
user avatar
  • 369
0 votes
0 answers
18 views

How to encode high dimensional dynamic categorical data?

Example: Facebook newsfeed ranking. This is typically done by relevance scoring each post using a regression model based on a number of features. Typical features might include information from the ...
user avatar
  • 101
0 votes
1 answer
32 views

Handling categorical data with more over 100 unique classes

I am working with a pure categorical data set. And some classes have more than 100 unique values. I could not find any appropriate encoding possibility. So I created a SQL table, where each value got ...
user avatar
  • 3
1 vote
1 answer
43 views

Categorical encoded variables in scikit-learn diabetes dataset [closed]

When using sklearn.datasets import load_diabetes, the sex variable which is categorical, is scaled to continuous values. Is it even legal to scale such variables?
user avatar
2 votes
2 answers
134 views

Is it acceptable to use label encoding for nominal categorical data when one hot encoding would create too many features?

I'm working on a short data science project to compare the accuracy of different classification methods. The groups decided to use and compare Random Forest, Naive Bayes and SVM. The dataset we are ...
user avatar
1 vote
2 answers
103 views

Different number of features in train vs test when using Label Encoding

This is not a duplicate of Different number of features in train vs test There are some categorical columns in my data, and the cardinality for each of them is large, so I chose to use LabelEncoding ...
user avatar
2 votes
1 answer
199 views

kNN for non-ordinal variables

kNN is a distance-based method, so it requires the input to be in numerical form. I was wondering if it is possible to use kNN imputer for non-ordinal categorical variables (like color). Since the ...
user avatar
2 votes
1 answer
66 views

Using the curse of dimensionality for encoding non-ordered (nominal) categorical variables of high cardinality

When the dimension is high, all data are approximately at the same distance away from each other. This makes distance-based methods such as k-nearest neighbors less useful if the data are more or less ...
user avatar
0 votes
0 answers
446 views

Tensorflow - I don't get the right shapes - `ValueError: Shapes (9, 1) and (8, 9) are incompatible`

I want to train a Sequential Neural Net (NN) with Tensorflow. ...
user avatar
3 votes
1 answer
50 views

One hot encoding of target variable containing classes 1 to 9 not including zero

While predicting a solution for a sudoku puzzle using CNN, the target variable should predict values from 1 to 9 for all the 81(9*9) values in the puzzle. Hence the target value shape is (81,9). Using ...
user avatar
7 votes
5 answers
2k views

How do I encode the categorical columns if there are more than 15 unique values?

I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have ...
user avatar
0 votes
1 answer
283 views

why should i do target encoding within cv loop?

i wish to use target encoding, using the category encoders sklearn library. I don't really understand why it is necessary to include this as a step in a sklearn pipeline WITHIN the cross validation ...
user avatar
  • 466
0 votes
1 answer
35 views

How to Present All Categories in All Samples

I have a data contains many categorical columns. When I sampled this data randomly a few times and applied one-hot encoding to categorical columns I noticed that it ended up with datasets with ...
user avatar
  • 422
2 votes
2 answers
567 views

Choose-many categorical features: alternatives to one-hot encoding?

I'm building a model to predict the lifetime value of a client based on the relational data we have on them. The user table has a bunch of one-to-many child tables that might be predictive. Grossly ...
user avatar
  • 123
0 votes
2 answers
39 views

Handling encoding of a dataset which has more than total 2000 columns

Whenever we have a dataset to be pre processed, before feeding it to the model we convert the categorical values to numerical values for which we generally use LabelEncoding, One Hot encoding etc ...
user avatar
  • 101
0 votes
1 answer
76 views

Encoding of high cardinality multi-label categorical feature?

This is the problem of binary classification: "1" - the subscriber is a driver (belongs to the segment of drivers), "0" - the subscriber is not a driver (does not belong to the ...
user avatar
1 vote
1 answer
380 views

Should one-hot encoded categorical features needs to be scaled when used along with text feature while deriving semantic similarity?

My aim is to derive textual similarity using multiple features. Some of the features are textual for which I am using (Tfhub 2.0) Universal Sentence encoder. There are other categorical features which ...
user avatar
  • 111
1 vote
0 answers
183 views

Cat2Vec implementation X = categorical and y = categorical

I am trying to convert categorical values (zipcodes) with Cat2Vec into a matrix which can be used as an input shape for categorical prediction of a target with binary values. After reading several ...
user avatar
  • 11
0 votes
2 answers
1k views

Encoding categorical data with pre-determined dictionary

in case feature encoding, if I'd like to encode my values based on my pre-determined dictionary, how do I do that? For instance, say, I've values as ...
user avatar
5 votes
1 answer
2k views

Target encoding with KFold cross-validation - how to transform test set?

Let's say I have a categorical feature (cat): ...
user avatar
  • 203
3 votes
0 answers
52 views

What is the concept behind the categorical-encoding used in the CatBoost benchmark problems?

I'm working through CatBoost quality benchmark problems (here). I'm particularly intrigued by the methodology adopted to convert categorical features to numerical values as described in the ...
user avatar
  • 161
1 vote
1 answer
142 views

One-hot & interaction one-hot on multiple categorical

I was wondering if there is any value to creating combined features out of multiple categorical variables when the individual categorical variables are already one-hot encoded? Simple example: there ...
user avatar
3 votes
1 answer
652 views

Encoding and cross-validation

Recently I've been thinking about the proper use of encoding within cross-validation scheme. The customarily advised way of encoding features is: Split the data into train and test (hold-out) set Fit ...
user avatar
  • 95
3 votes
1 answer
247 views

On gradient boosting and types of encodings

I am having a look at this material and I have found the following statement: For this class of models [Gradient Boosting Machine algorithms] [...] it is both safe and significantly more ...
user avatar
1 vote
1 answer
684 views

What is sum encoding?

I've heard sum encoding mentioned as a method of encoding categorical variables, but I haven't been able to find a clear explanation of what it actually is. I found the following explanation on ...
user avatar
  • 113
2 votes
2 answers
1k views

Why is count encoding effective in improving accuracy? [duplicate]

Can someone please explain why/how Count encoding of categorical features improve accuracy in classification when compared to simply label encoding them ? I found one explanation in kaggle " ...
user avatar
  • 157
0 votes
1 answer
76 views

Is this attribute numeric or categorical (ordinal)? Help!

So I have this dataset I need to perform several techniques on as part of a data mining/machine learning project of some sort in PYTHON. There are a couple of features however, that have me very ...
user avatar
  • 55
0 votes
2 answers
101 views

Categorical and non-categorical data in the same column

I have a unique dataset that has many columns and most columns contain both categorical and non-categorical data. For example, let's say that one column is attribute_1 and for observations that have ...
user avatar