Questions tagged [encoding]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2
votes
1answer
19 views

How to work with different Encoding for Foreign Languages

I've got a Word Embedding File called model.txt. This contains 100 Dimensional vectors for over a million French words. These words contain accented characters such ...
1
vote
1answer
25 views

File path encoding to feature

I am trying to find some sort of encoding algorithm that would allow to transform system file paths eg. "c:/users/file1/subfile2/targetfile" into a feature that I could use in machine ...
0
votes
1answer
17 views

ELMo - How does the model transfer its learning/weights on new sentences

Word2vec and Glove embeddings have the same vector representation for every word in the corpus and does not take context into consideration. For eg: The dog does bark at people The bark of the tree ...
0
votes
3answers
39 views

How do i get dummies for this dataset

I am using an udemy course for MachineLearning and I am trying to form a dummy for my variable the column is Country I want to change to France Germany Spain France ...
0
votes
0answers
101 views

Encoding with OrdinalEncoder : how to give levels as user input?

I am trying to do ordinal encoding using: from sklearn.preprocessing import OrdinalEncoder I will try to explain my problem with a simple dataset. ...
0
votes
0answers
51 views

What is “position” in CNN (im2latex) for Positional Encoding?

I'm trying to build a model that maps images of math formulas into LaTeX markup. I found an acticle (https://arxiv.org/ftp/arxiv/papers/1908/1908.11415.pdf) that proposes an encoder-decoder ...
0
votes
0answers
14 views

Target Encoding and Feature Scaling

I am using Support Vector Classification which performs well when we have done Feature Scaling, however, I am using Target Encoding on my categorical variables. Is it advisable to do Feature Scaling ...
0
votes
0answers
70 views

Mean encoding With KFold regularization

I just learned that regularizing the mean encoding reduce the leakage hence generalize better than mean encoding without it but I made 2 submissions with XGB in <...
0
votes
1answer
52 views

Aggregating multiple encoded categorical values

I am trying find commonly used techniques when dealing with high cardinality multi-valued categorical variables. I am currently using a dataset with a feature CATEGORY which has a cardinality of ~20,...
0
votes
1answer
34 views

How to do target encoding when data has repeated rows?

How can I do encoding for a category when data has repeated rows? Can I do target encoding? Or Is there another encoding I can use? I want to figure how to include a categorical variable in a model ...
0
votes
0answers
27 views

One Hot Encoding where all sequences don't have all values

Is there a way (other than manually creating dictionaries) to one hot encode sequences in which not all values can be present in a sequence? sklearn's OneHotEncoder ...
0
votes
0answers
9 views

Can i use different types of encodings for categorical variables in one dataset

Should I mix encodings. For example for features age and income i have one type of encoding and for features typeOfPerson i have another?
2
votes
1answer
18 views

Memory efficient encoding logic for group categories

I have a huge dataset with categorical data. It is comprised of alerts having multiple properties. Each alert belongs to a group, and some even belong to multiple groups. It looks somewhat like this: ...
1
vote
1answer
19 views

LabelEncoder with a Multi-Layer Perceptron?

So we're working on a machine learning project at work and it's the first time I'm working with an actual team on this. I got pretty good results with a model that uses the following SKLearn pipeline: ...
1
vote
1answer
29 views

How to encode a column containing both string and numbers

I have a column in my dataset which contains both number and strings as the value. I want to encode the string variable so to use it for predicting. What is the best way to do this?
2
votes
1answer
24 views

Magnifying or reducing the size of input groups into a neural network

Say you've got two inputs (X1 and X2) that you want to use to predict Y. You're not sure how important X1 and X2 are for predicting Y, but you assume about even. One-hot encoding is a good strategy ...
0
votes
1answer
167 views

splitting into train test by train_test_split of float values?

How to split into train test by train_test_split of float values ? I used LabelEncoder but I have about 300K lines and when I used the cross_val I saw ...
2
votes
1answer
24 views

Different encoders applied to a dataset

I have a dataset which have both categorical features with high cardinality (>8000) and low cardinality (4 or 5). Would that be ok to encode the high cardinality ones with one encoder (target encoder,...
1
vote
2answers
103 views

Unsupervised encoding of categorical features

I have multiple log records with discrete categorical features. Shape of my dataset is (100k, 24) My aim is to look for anomalies in these records. I am planning to cluster the data after encoding. ...
1
vote
1answer
24 views

Validity of PU learning while using character-level encoding using CNNs for classifying text data

I'm trying to classify a large set of documents (~100M) as valid or invalid, based upon a small given set of labeled valid documents (~3k). I'd like to know if the PU learning approach described in ...
10
votes
1answer
333 views

Why does frequency encoding work?

Frequency encoding is a widely used technique in Kaggle competitions, and many times proves to be a very reasonable way of dealing with categorical features with high cardinality. I really don't ...
0
votes
2answers
468 views

Binary Classification - One Hot Encoding preventing me using Test Set [duplicate]

I have a preprocessing pipeline that includes replacing missing values and onehotencoding for the categorical variables. When I try to use my model on the test set, it explains that the number of ...
0
votes
2answers
59 views

Encoding Categorical Data Without Increasing the Dimension

I've been exploring methods for encoding categorical data. I was hoping to find a good method that does not increase the dimension of the dataset, similar to the one used on this dataset about drug ...
2
votes
1answer
53 views

Could I add a one hot encoding to each feature representing “has data” versus “has no data”

I have a data set that has some holes in it. I was wondering if I could add two columns for each feature representing this feature has data and ...
1
vote
2answers
112 views

Is it possible to know the output vectors of MLP Classifier of scikit learn?

I'm a beginner with scikiti-learn library. I have an ANN with 3 input, 2 hidden layers and 3 output. ...
1
vote
2answers
774 views

What is the advantage of positional encoding over one hot encoding in a transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, they used positional encoding with sin for even indices and cos for odd indices. In the paper (Section 3.5), they ...
0
votes
1answer
392 views

Frequency/Count encoding

How do I perform frequency/count encoding for a train and test set? The implementations of this encoding I've seen simply frequency encode the categorical variables on a particular dataset (no ...
0
votes
1answer
94 views

Target encoding before split on skewed data

Hi My data is distributed like follow: And I only have categorical variables on many many levels. As I need to make a regression task I thought about doing leave one out encoding on my categories. ...
1
vote
2answers
40 views

Does the predict function in machine learning understand categorical data

I understand that before feature engineering one has to split the dataset into train and test data, so as to avoid bias in the analysis. I also understand that the machine learning model does not ...
0
votes
0answers
26 views

Mean encoding and covariate shift

I am working on a binary classification problem whose two main issues are categorical variables with many levels and, since the process, I am modeling is not yet stable, the proportion among the two ...
0
votes
0answers
15 views

How to encode factor predictors in prediction models

The response variable as well as all predictor variables in my dataset are factors. I want to build a model for predicting the response variable. As I understand I have to first encode my predictor ...
0
votes
0answers
7 views

Include or exclude original features after encoding

I have some categorical features and they are encoded by different types of encoding (one-hot, label, target, etc). My question is whether you usually include the original categorical features with ...
1
vote
1answer
73 views

Encode missing data and unseen data

Let's assume that I have a classification problem and all my features are categorical data. I have missing data (and I do not want to do any imputation). Also, I know that I will have some unseen ...
0
votes
0answers
28 views

One hot encoding of continuous floats using keras backend

I am trying to convert a vector of floats into a matrix similar to one hot encoding, but I want this to happen in non-discrete space to retain gradients. Therefore I am only able to use keras backend ...
0
votes
0answers
10 views

How to encode features that encode regular values as well as special categorical values

I was recently playing around with the FICO explainable machine learning challenge dataset. In the dataset, there are a bunch of numerical features which have values values typically in the 0-100 ...
3
votes
0answers
395 views

Target mean encoding worse than ordinal encoding with GBDT ( XGBoost, CatBoost )

I have a dataset of 23k rows of an unbalanced dataset 85/15 ratio, 10 variables ( 9 of which are categorical ) , i'm using CatBoost and XGBoost for a binary classification. I applied cv (5 iteration ...
0
votes
0answers
63 views

Using OrdinalEncoder on entire Dataframe and Avoiding Header transformation?

I want to encode the string in a dataframe which also has float values (which I don't want to change). The each feature column has several different unique strings. ...
1
vote
1answer
567 views

String handling by OneHotEncoder

I am reading everywhere on new questions and blogs that since version 0.20, OneHotEncoder is able to handle string features. Moreover, the documentation is what looks more ambiguous. Here are the ...
2
votes
3answers
360 views

One hot encoding with too many features (~ 10,000)

I am building a model to predict time off and sick leave for a specific employee. Each of the employees has one row per day from 01/01/2013 to 31/12/2018 in the dataset flagged with 0 or 1 (if that ...
2
votes
1answer
37 views

How can I count the number of occurrences of a category in dataset as part of an Sklearn Pipeline

Let us say we have a dataset with a feature such as Surname. arr['Surname'] = ['Smith', 'Jones', 'Johnson', 'Smith'] And I want to encode this categorical info ...
0
votes
0answers
15 views

Not able to sentence encode a list of sentences using multiprocessing technique - pool.map() function in python

I am trying to embed a text data which is in the form of list, since its a huge data I wanted to embed it using the multiprocessing Pool map() function. The embedding technique I'm using is google's ...
6
votes
2answers
681 views

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

While reviewing the Transformer architecture, I realized something I didn't expect, which is that : the positional encoding is summed to the word embeddings rather than concatenated to it. ...
0
votes
0answers
22 views

Is random forest a kind of spatial feature encoding?

From the book "Deep Learning and Convolutional Neural Networks for Medical Image Computing" On the other hand, CNN models have been proved to have much higher modeling capacity, compared to the ...
2
votes
0answers
47 views

Why don't Target/LeaveOneOut Encoders work well for Regression Tasks?

In this review of categorical encoding, it states early on that For regression tasks, Target and LeaveOneOut probably won’t work well and later repeats that Target/LeaveOneOut (Owen Zhang's ...
5
votes
1answer
88 views

Mapping of categorical features into binary indicator features

I am currently reading an introductory machine learning book by Daumé (ch. 03, p. 30) and when discussing the mapping of categorical features with "n" possible values into "n" binary indicator ...
0
votes
1answer
127 views

String indices must be integers

I was trying to encode the string values of the feature 'ProductCategory' into integer values but I got this error. Kindly help. And I would also like to ask if label-encoding this feature would not ...
0
votes
0answers
149 views

Pre-processing data to make predictions on deployed Sklearn model

I am new to Machine Learning. I have trained a ML model on the Diamond Prices Dataset to predict the price of a diamond given it's features (carat, cut color, clarity, etc...) I have used pickle to ...
0
votes
1answer
432 views

How does Byte Pair Encoding work?

I am using this to do some Byte Pair Encoding (BPE). My corpus looks like this. When I run the learn_bpe, I get a vocabulary that looks like this. ...
4
votes
1answer
92 views

How to automate the encoding process?

I am working on the Boston challenge hosted on Kaggle and I'm still refining my features. Looking at the dataset, I realize that some columns need to be encoded in binary, some encoded in decimals (...