2022 Developer Survey is open! Take survey.

Questions tagged [encoding]

Encoding in machine learning and data science refers to the process by which non-numeric data is transformed into a numeric representation that can be fed into machine learning algorithms.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

What is the difference between one hot encoding and 1-of-c encoding?

I am tasked with using 1-of-c encoding for a NN problem but I cannot find an explanation of what it is. Everything I have read sounds like it is the same as one hot encoding... Thanks
user avatar
  • 111
0 votes
0 answers
32 views

Do I need to encode numerical variables like "year"?

I have a simple time-series dataset. it has a date-time feature column. ...
user avatar
0 votes
0 answers
16 views

Can anyone tell me why is my pipeline wrong?

I am trying to build a pipeline in order to perform GridSearchCV to find the best parameters. I already split the data into train and validation and have the following code: ...
user avatar
1 vote
1 answer
24 views

Is it vital to do label encoding with target variable

Should I always use label encoding while doing binary classification?
user avatar
0 votes
1 answer
21 views

Should I encode the categorical data before making a training validation split?

I am looking at some examples in kaggle and I'm not sure what is the correct approach. If I split the training data for training and validation and only encode the categorical data in the training ...
user avatar
0 votes
0 answers
6 views

Encode record with no exemptions in clustering (standardization)

I am trying to apply clustering, k-means, to a custom dataset I have created. There are several features and most have exemptions (min/max dollar amount). There are some records, however, that do not ...
user avatar
1 vote
1 answer
66 views

Converting binary valued data to boolean or one hot?

I am dealing with a dataset that contains multiple columns (features) that contain binary variables, e.g., a gender feature that contains 'male' and 'female'. I want to apply some supervised learning ...
user avatar
  • 73
0 votes
1 answer
22 views

What's the minimum percentage of categories should be present in the categorical variable for to ignore the variable entirely

For example, if i have a feature "colour_codes" that has close to 5000 distinct color codes inside it. And the number of samples/rows is 10 million. Then should I ignore the feature "...
user avatar
0 votes
0 answers
16 views

Best loss function for baseN encoding LSTM model

everyone, I am trying to train an LSTM model for sequence prediction. I have X and Y which are two numpy arrays. X is a list of integers (integer encoded strings) while to encode Y I used a BaseN ...
user avatar
1 vote
1 answer
56 views

Categorical feature encoding

I am making a classification model. I have categorical and continuous data. The categorical columns include columns with 2 classes such as sex (male, female), and multi-class columns such as location. ...
user avatar
  • 11
0 votes
1 answer
30 views

Retrieving the ordinal encoding of a variable after it's placed in a pipeline/columntransformer

I am applying ordinal encoding to a dataset through a column transformer - how can I retrieve the ordinal encoding of a feature (e.g. Area)? ...
user avatar
0 votes
1 answer
100 views

Getting dummies for both train and test data

Should I apply pd.get_dummies() for both train and test data? And would it not result in data leakage?
user avatar
0 votes
0 answers
15 views

Are there differences in preprocessing nominal vs ordinal vs interval vs ratio data

I wonder are there significant differences that ought to be known when preprocessing nominal vs ordinal vs interval vs ratio. Intuitively, it seems like encoding ordinal values should be performed ...
user avatar
2 votes
1 answer
69 views

Encode a set of skills into a feature

I am working with a dataset where users have a set of skills. I have more than 500 skills and I was wondering what is the best way of encoding a vector, e.g., ...
user avatar
  • 143
0 votes
0 answers
27 views

Dealing with discrete variables as continuous for K-means clustering (or not)

It is well established that k-means works best, and is designed for, continuous variables. I am considering a clustering problem where I have data like this: total spend / $ number of items in basket ...
user avatar
1 vote
0 answers
5 views

quereies related to autoencoder

i want to design an deep auto encoder after following keras tutorial. Input is a simple 2-dimensional image consists of 512 rows and 50 columns matrix My trial code is ...
user avatar
0 votes
0 answers
97 views

Expected input for time-series transformers

I am trying to create a model starting from the Attention is all you need paper. Specifically I want to setup the Encoder/Decoder architecture to predict time-series. I would like to implement it in ...
user avatar
  • 111
1 vote
3 answers
603 views

One-hot-encoded variables dominating clustering

I am performing some unsupervised clustering with k-means on some transaction data that contains the following information: Customer units purchased in category_1 units purchased in category_1 time ...
user avatar
0 votes
0 answers
21 views

Encoding categories of items purchased for customer segmentation (clustering)

AIM: I am trying to deveop a model that will allow me to understand my customer better by clustering their purchase behaviour. CONTEXT: I have transaction level data that tells me, for a given ...
user avatar
1 vote
1 answer
12 views

Encoding distance variable that is continuous until out-of-range

I have a varaible distance which is continous until a "hard stop" at which we stop measuring the distance itself and just label the distance as "out ...
user avatar
0 votes
1 answer
193 views

Encoding with OrdinalEncoder: TypeError: unhashable type: 'numpy.ndarray'

I am trying to do a Random Forest in a dataset with numerical and categorical variables in order to obtain a categorical result (two possible classes, column name "predicción"). I am using ...
user avatar
0 votes
2 answers
27 views

Encoding feature containing both text and string?

I have a feature which has following entries:- | Exterior | | -------- | | Vinyl | | Wd Sdng | | MetalSd | | Wd Sdng | | HdBoard | | BrkFace | | Wd Sdng | ...
user avatar
  • 1,241
2 votes
3 answers
136 views

Does One-Hot encoding increase the dimensionality and sparsity of dataset?

There are two ways to convert object datatype into numeric datatype, first is One-Hot encoding and second is simply map the numerical tags to different values. For example for column Age containing ...
user avatar
1 vote
2 answers
52 views

Encoding "histogram bins"

I am currently working on a regression problem where I have one variable (x) of the data in the form of "histogram bins". I.e. I could have value ranges ...
user avatar
0 votes
0 answers
7 views

Best Approach to Encode/Categorize Unrelated Data

I have a data set with multiple columns that I'd like to encode for a model. The data set is as follows. Transaction CPT Code Modifier1 Modifier2 Modifier3 Price 00001 99287 LT 81 50.00 00002 ...
user avatar
0 votes
1 answer
40 views

Encoding when different number of records for each month-One hot or different type?

I am working on a dataset which is pretty small: 1169 records. There is a column called month and it takes the values 'Jan','Feb', or 'March'. The number of records for each month is different. I have ...
user avatar
0 votes
0 answers
27 views

Encoding technique used in Keras ImageDataGenrator class

I would like to know what encoding is used by ImageDataGenerator for encoding the class labels. I have done a lot of research and found that there is a variable called ...
user avatar
  • 7
6 votes
3 answers
1k views

What is the difference between one-hot and dummy encoding?

I am trying to understand The reason behind encoding (one-hot encoding and dummy encoding) How one-hot and dummy are different from each other
user avatar
0 votes
0 answers
16 views

Dataset that's already encoded, mixed types, methods for EDA and clustering?

I have a dataset that is already encoded (so it looks like table 2) with numbers (ie. each state & flavor have an assigned number, likeliness to try new flavor is on a scale 1-5, etc.) I would ...
user avatar
  • 101
0 votes
0 answers
16 views

Term document matrix for webpages

Consider the following code for obtaining term-document matrix for given texts ...
user avatar
  • 147
0 votes
0 answers
11 views

How to apply Helmert Coding in a real Machine Learning model?

My dataset is something like this ...
user avatar
  • 497
0 votes
1 answer
32 views

Preprocess multi-sample time series data: encode each sample separately or in aggregate?

Let's say I have 3 dense sequences of uniform length. Should I fit a scaler on them separately or together? ...
user avatar
  • 476
0 votes
2 answers
92 views

How to encode a sentence using an attention mechanism?

Recently, I read about one of the state-of-the-art method called Attention models. This method use a Encoder-Decoder model. It can find a better encoding for each word in a sentence. But how can I ...
user avatar
2 votes
2 answers
219 views

How to handle non ordinal Features like Gender,Language,Region etc? Ordinal Encoding or one-hot encoding?

I see that usually, while preparing the dataset. Usually, data scientists convert non-ordinal features like Gender or Language in a dataset using LabelEncoder/ordinalEncoder. Ideally, they should have ...
user avatar
1 vote
0 answers
34 views

When doing feature selection, are features like "year", "month" considered as ordinal features or should I convert them to strings?

I am working on a hotel reservation dataset that has both categorical and numerical (continuous and discrete) features (26 columns, 30244 rows). Target is also categorical and it says if the user &...
user avatar
2 votes
1 answer
1k views

Should I do one hot encoding before feature selection and how should I perform feature selection on a dataset with both categorical and numerical data

a newbie here. I am currently self-learning data science. I am working on a dataset that has both categorical and numerical (continuous and discrete) features (26 columns, 30244 rows). Target is ...
user avatar
0 votes
1 answer
34 views

Handling categorical data with more over 100 unique classes

I am working with a pure categorical data set. And some classes have more than 100 unique values. I could not find any appropriate encoding possibility. So I created a SQL table, where each value got ...
user avatar
  • 3
0 votes
1 answer
21 views

Do I need to encode samples during inference?

I recently started saving (pickling) my fitted encoders. The thinking was that I would need them to encode previously unseen samples during inference. Encode training features and labels. Train model ...
user avatar
  • 476
0 votes
3 answers
420 views

encoding datetime features .. label or onehot?

I am working on a cab booking prediction problem where I need to use datetime aspects like hour,day ,week etc for prediction. As I need to do categorical encoding for the purpose. can anyone help me ...
user avatar
2 votes
2 answers
137 views

Is it acceptable to use label encoding for nominal categorical data when one hot encoding would create too many features?

I'm working on a short data science project to compare the accuracy of different classification methods. The groups decided to use and compare Random Forest, Naive Bayes and SVM. The dataset we are ...
user avatar
0 votes
1 answer
51 views

How to do encode this target vector containing strings

Consider a target vector like ...
user avatar
1 vote
2 answers
61 views

Linear Regression with Category variables

I'm currently learning and exploring machine learning and understand the basics of linear regression based on two numerical variables, but now I wish to go a little further and need some guidance ...
user avatar
  • 111
0 votes
1 answer
63 views

Encode the days of week as numeric variable

I would like to understand if there is the possibility to encode the days of the week as a single numerical column to preserve the ordinal relationship between the days. My task is a classification ...
user avatar
  • 193
2 votes
1 answer
232 views

Multi-Feature One-Hot-Encoder with varying amount of feature instances

Let's assume we have data instances like this: ...
user avatar
1 vote
0 answers
25 views

Modeling time spent on different websites

Apologies in advance for the poor title. I didn't know exactly how to phrase what I want in a succinct manner so hopefully I can elaborate a bit more. I have a dataset where I have customer_id, ...
user avatar
  • 111
0 votes
1 answer
24 views

Cleaning NaNs with averages pre or post split? [duplicate]

I have a column with some NaNs in it and I want to replace those NaNs with the average/median/mode. Technically, the validation/ test data has never been seen before - so how could I include it in the ...
user avatar
  • 476
7 votes
5 answers
2k views

How do I encode the categorical columns if there are more than 15 unique values?

I'm trying to use this data to make a data analysis report using regression. Since regression only allows for numerical types, I then need to encode the categorical data. However, most of these have ...
user avatar
2 votes
2 answers
573 views

Choose-many categorical features: alternatives to one-hot encoding?

I'm building a model to predict the lifetime value of a client based on the relational data we have on them. The user table has a bunch of one-to-many child tables that might be predictive. Grossly ...
user avatar
  • 123
0 votes
1 answer
34 views

How would I approach training a model and encoding this categorical data

So I have the following data: I have one series where each word has a value that describes the average review score that would get. For example, if the word "excellent" showed up in reviews ...
user avatar
1 vote
1 answer
32 views

Encoding Tags for Random Forest

I have the following data set: I want to use attributes Tags and Authors to classify each record into their respective Rating. In order to do so I want to use a random forest classifier. My concern ...
user avatar
  • 13