Questions tagged [encoding]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
6
votes
1answer
91 views

Mapping of categorical features into binary indicator features

I am currently reading an introductory machine learning book by Daumé (ch. 03, p. 30) and when discussing the mapping of categorical features with "n" possible values into "n" binary indicator ...
0
votes
1answer
134 views

String indices must be integers

I was trying to encode the string values of the feature 'ProductCategory' into integer values but I got this error. Kindly help. And I would also like to ask if label-encoding this feature would not ...
0
votes
0answers
154 views

Pre-processing data to make predictions on deployed Sklearn model

I am new to Machine Learning. I have trained a ML model on the Diamond Prices Dataset to predict the price of a diamond given it's features (carat, cut color, clarity, etc...) I have used pickle to ...
0
votes
1answer
435 views

How does Byte Pair Encoding work?

I am using this to do some Byte Pair Encoding (BPE). My corpus looks like this. When I run the learn_bpe, I get a vocabulary that looks like this. ...
4
votes
1answer
96 views

How to automate the encoding process?

I am working on the Boston challenge hosted on Kaggle and I'm still refining my features. Looking at the dataset, I realize that some columns need to be encoded in binary, some encoded in decimals (...
0
votes
1answer
73 views

Binary Encoding of Ordinal Categories

I have a data frame in which one of my columns is the target value and there are lots of ordinal categories in columns of data frame. I want to encode these ordinal categories in columns with this ...
1
vote
3answers
483 views

Is it right to impute Train and Test set?

I am experimenting with a dataset and I have a couple of columns with high cardinality. So, I performed mean target encoding (given that my dataset had more than 50000 observations). But, before doing ...
2
votes
2answers
2k views

Applying mean encoding before or after splitting into train and test set

I have a dataset of 50000 observations with columns of high cardinality. The best way to encode them is with mean encoding, then to use regularization. I will use CV rather than smoothing. But when I ...
30
votes
3answers
19k views

What is the positional encoding in the transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, there is a picture: I don't know what positional encoding is. by listening to some youtube videos I've found out that ...
3
votes
2answers
76 views

why One-Hot Encoder can avoid the situation that the model will misunderstand the data to be in some kind of order if the data has been Label Encoding

We know that we prefer to using One-Hot Encoding not Label Encoding when processing with non-ordinal data. And I real a blog which give the difference between Label Encoding and One-Hot Encoding. So ...
1
vote
0answers
52 views

Aggregating target-encoded array-like categorical features?

I am trying find commonly used techniques when dealing with high cardinality multi-valued categorical variables for machine learning classification algorithms. One-hot encoding leads to very high ...
4
votes
1answer
2k views

How to handle columns with categorical data and many unique values

I have a column with categorical data with nunique 3349 values, in a 18000k row dataset, which represent cities of the world. I also have another column with 145 nunique values that I could also use ...
4
votes
2answers
2k views

In which cases shouldn't we drop the first level of categorical variables?

Beginner in machine learning, I'm looking into the one-hot encoding concept. Unlike in statistics when you always want to drop the first level to have k-1 dummies (...
3
votes
1answer
403 views

Target Encoding: missing value imputation before or after encoding

I want to perform a target encoding for my categorical features although I am not sure when to perform the data imputation if any of them has missing values. Let's say I have a few continuous features,...
0
votes
1answer
54 views

How to deal with name strings in large data sets for ML?

My data set contains multiple columns with first name, last name, etc. I want to use a classifier model such as Isolation Forest later. Some word embedding techniques were used for longer text ...
3
votes
1answer
2k views

One-hot encode multi-class multi-label sequences

Suppose I want to build a timeseries where each timestep is represented by a categorical array: the encoded sequences look like [[2, 0, 5],[3, 1, 4],..] and each ...
3
votes
1answer
655 views

For a multi-class classification problem, how to transform the target variable to a form that is usable by sklearn algorithms?

I recently tried to create a model for predicting what class a sample belongs to out of 160 possible classes. These classes of the target variable are just simple strings describing workouts like "...
2
votes
1answer
1k views

Target encoding with cross validation

I am trying to understand this way of target (mean/impact/likelihood) encoding using (two-level) cross validation. It's taking mean value of y. But not plain mean, but in cross-validation within ...
1
vote
0answers
113 views

How to encode H3 geohash in regression model

I'm trying to train a random forest regression model based on a number of features, including location. I know that raw lat/long can't be used directly, so I've bucketed them using H3. I'm struggling ...
2
votes
0answers
113 views

how to extract the Top contributing labels/words in universal-sentence-encoder-large - TransformerModel?

I'm using the universal-sentence-encoder-large (Transformer Model) encoding process for embedding and then using the embedding for Clustering - Basically for unsupervised learning. I want to get the ...
1
vote
1answer
1k views

Scaling label encoded values for Linear Algorithms

I have encoded categorical variables to numerical values. As we know that for feeding values to Linear Algorithms like SVM or KNN, we scale the values for columns having large variations. I have ...
1
vote
1answer
311 views

Encoding multiple observations from the same feature space

My data contains multiple observations of categorical feature.The feature space is medical symptoms, so the data for this feature is like : ['fever','pain','yellow skin' .... ] .The amount of symptoms ...
4
votes
2answers
2k views

How to use one hot encoding of string categorical features in keras?

I am dealing with a binary classification problem. The output column of my dataset is already encoded in 0/1. The problem is that I have many categorical features (columns), which are strings and I ...
1
vote
3answers
258 views

One Hot Encoding of Age

My task is to predict how many years a person has left to live using an MLP. There is one specific feature I'd like to discuss: current age. Statistically, it's a conditional probability. Example: ...
37
votes
2answers
22k views

Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)

Which is better for accuracy or are they the same? Of course, if you use categorical_crossentropy you use one hot encoding, and if you use sparse_categorical_crossentropy you encode as normal integers....
2
votes
3answers
521 views

Transformation of categorical variables (binary vs numerical)

When using categorical encoding, I see some authors use arbitrary numerical transformation while others use binary transformation. For example, if I have a feature vector with values A, B and c. The ...
21
votes
3answers
9k views

Difference between OrdinalEncoder and LabelEncoder

I was going through the official documentation of scikit-learn learn after going through a book on ML and came across the following thing: In the Documentation it is given about ...
0
votes
1answer
939 views

Should I use pandas get_dummies and create additional columns or use my own encoding code that keeps 1 column?

I am running the Kaggle Video Games sales dataset through an XGboost algo. I want to encode the categorical column of "Game Rating" from E, M, etc. to 0-5 when I use: data = pd.get_dummies(data=...
2
votes
4answers
70 views

Equivalent of numeric encoding when rows can contain multiple values

If we have a column like: Name 0 Alice 1 Bob 2 Dave then, after numeric encoding, it becomes: ...
2
votes
0answers
142 views

“Binary Encoding” in “Decision Tree” / “Random Forest” Algorithms

Is it OK to use Binary Encoding in a dataset containing categorical columns with very high cardinalities? Some facts about my dataset: My dataset has ~170000 rows One of the categoric variables has ...
1
vote
0answers
10 views

What happens if I do not encode the lables or classifiers in the data? [closed]

I have a data where the three variables are numerical and one variable is a string. I am using the simple decision tree algorithm. Read somewhere that the data in strings must be encoded with one hot ...
5
votes
2answers
5k views

Muti-hot encoding vs Label-Encoding

I am learning about different input-vector representations for Neural Networks One of the alternatives to sparse One-Hot encoded vector is the Multi-Hot encoding. Do I understand correctly that a ...
0
votes
1answer
28 views

How to transform dictionary data into a string vector?

I have key,value data where each record is in a Python string. An example record looks like this: ...
-2
votes
1answer
2k views

How to use same encode label with same value used in training

I did save my model and using that model I want to predict the data. I am using Flask HTTP server for prediction endpoint. I have training data like this. I did save my model and using that model I ...
2
votes
2answers
99 views

How to encode data with a feature having multidimensional features (colors)?

My dataset has around 20 features, one of which is colors(in string format). There are around 50 different colors. I have converted them to RGB, but now I want to encode the data in such a way that ...
0
votes
1answer
371 views

How to use multiple encoders(one-hot and numerical) together for PCA

I want to implement PCA on a dataset(retail) but the data is categorical. One-hot encoding on some columns like Gender, Fabric, Brand makes sense but on other features like price range, size, I would ...
3
votes
2answers
2k views

One hot encoding large dataset

Initially, I have a dataset where for each row there is user_id and product_ids he bought. In that dataset there are 478191 products bought by different users. In order to discover frequent items ...
1
vote
0answers
36 views

How to pre-process frequency of a series of signals?

I use a neural net to generate predictions based on a time series of signals. I use a sliding window to feed the data to an LSTM model. The input signals have a random frequency that - I believe - is ...
2
votes
1answer
4k views

One hot encoding at character level with Keras

I am reading Chollet's book on deep learning at the moment and in the NLP chapter he says: ...
-2
votes
1answer
823 views

Encoding categorical data 2 different columns

Suppose I have two columns namely Goods, and Quality which are to be one encoded. Goods                   &...
7
votes
2answers
11k views

One Hot Encoding vs Word Embeding - When to choose one or another?

A colleague of mine is having an interesting situation, he has quite a large set of possibilities for a defined categorical feature (+/- 300 different values) The usual data science approach would be ...
2
votes
1answer
3k views

One hot encoding vs Word embedding

I am very confused between one hot encoding and word embedding in terms of structure of the network and how it reduces the dimensionality. I am currently using encog with c# which has some ...
4
votes
1answer
3k views

Is it effective to use one-hot encoding when its dimension is as large as thousands

Here I try to construct a classifier using DNN(deep neural network) with its inputs being many portfolios. In essence, each portfolio contains several stocks which are labeled by there inner-code, for ...
0
votes
1answer
755 views

Is it good practice to always remove highly correlated variables?

1- Would it always be beneficial to remove highly correlated features prior to training a model? If not, why not. 2- Would you perform One Hot encoding where applicable, prior to removing highly ...
5
votes
1answer
13k views

Always drop the first column after performing One Hot Encoding?

Since one of the columns can be generated completely from the others, and hence retaining this extra column does not add any new information for the modelling process, would it be good practice to ...
1
vote
1answer
1k views

Differences of get_dummies and labelbinarizer?

Are there any differences between get_dummies and labelbinarizer in terms of what they want to achieve? It seems to be both will somehow do a one-hot encoding.
1
vote
2answers
3k views

Faced problem while applying OneHotEncoder

For classification, I was trying to convert categorical data into numeric by applying OneHotEncoder. But it shows error could not convert string to float Here is ...
0
votes
1answer
725 views

One hot encoding error “sort.list(y)…”

I'm trying to do one hot encoding on a data set containing 4 categorical features in R. Unique levels per feature: 400, 60, 6, 5, respectively. I get the following error during the first call to the ...
2
votes
2answers
106 views

What approach for creating a multi-classification model based on all categorical features (1 with 5,000 levels)?

I have a data set I'm trying to create a predictor model for. The 5 features and outcome are all categorical data. One of the features contains 5,000 unique levels. While the other 4 are all under ...
13
votes
4answers
16k views

One hot encoding alternatives for large categorical values?

Hi have dataframe with large categorical values over 1600 categories is there any way I can find alternatives so that I don't have over 1600 columns. I found this below interesting link http://...