66
votes
Accepted
Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)
Use sparse categorical crossentropy when your classes are mutually exclusive (e.g. when each sample belongs exactly to one class) and categorical crossentropy when one sample can have multiple classes ...
- 776
66
votes
What is the positional encoding in the transformer model?
For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be
$$...
- 9,147
54
votes
Accepted
What is the positional encoding in the transformer model?
Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations:
Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings
Taking ...
- 656
47
votes
Difference between OrdinalEncoder and LabelEncoder
Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while ...
- 806
38
votes
Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)
The answer, in a nutshell
If your targets are one-hot encoded, use categorical_crossentropy.
Examples of one-hot encodings:
...
- 381
33
votes
Accepted
Encoding features like month and hour as categorial or numeric?
Have you considered adding the (sine, cosine) transformation of the time of day variable? This will ensure that the 0 and 23 hour for example are close to each other, thus allowing the cyclical nature ...
- 466
26
votes
Accepted
One Hot Encoding vs Word Embedding - When to choose one or another?
One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each ...
- 20.2k
26
votes
Difference between OrdinalEncoder and LabelEncoder
As for differences in OrdinalEncoder and LabelEncoder implementation, the accepted answer mentions the shape of the data:
...
- 393
19
votes
Accepted
How to deal with string labels in multi-class classification with keras?
Sklearn's LabelEncoder module finds all classes and assigns each a numeric id starting from 0. This means that whatever your class representations are in the ...
- 28.4k
18
votes
Encoding features like month and hour as categorial or numeric?
The answer depends on the kind of relationships that you want to represent between the time feature, and the target variable.
If you encode time as numeric, then you are imposing certain restrictions ...
- 641
15
votes
Accepted
Encoding with OrdinalEncoder : how to give levels as user input?
I'm not sure if you ever figured this out but I was trying to find answers on this exact same question and there aren't really any good answers in my opinion. I finally figured it out though.
...
- 166
14
votes
Accepted
One hot encoding alternatives for large categorical values
One option is to map rare values to 'other'. This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power.
I have also ...
- 2,238
14
votes
What is the positional encoding in the transformer model?
Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle).
But you have to take ...
- 2,262
9
votes
Accepted
In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?
When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have ...
- 22.3k
8
votes
Encoding features like month and hour as categorial or numeric?
I recommend using numerical features. Using categorical features essentially means that you don't consider distance between two categories as relevant (e.g. category 1 is as close to category 2 as it ...
8
votes
Accepted
How to handle columns with categorical data and many unique values
For categorical columns, you have two options :
Entity Embeddings
One Hot Vector
For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change ...
- 2,239
8
votes
What is the positional encoding in the transformer model?
To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think). They did not come up with the encoding.
Here is the PE lookup table generation ...
- 81
8
votes
Why does frequency encoding work?
Check this post.
In the cases where the frequency is related somewhat with the target variable, it helps the model to understand and assign the weight in direct and inverse proportion, depending on ...
- 814
8
votes
Accepted
What is the difference between one-hot and dummy encoding?
Most machine learning models accept only numerical variables. This is the reason behind why categorical variables are converted to number so the model can understand better.
Now lets address your ...
- 1,189
7
votes
Accepted
Pandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)
One advantage of get_dummies is that it can operate on values other than integers (so you don't need the LabelEncoder) and ...
oW_♦
- 6,254
6
votes
One Hot Encoding vs Word Embedding - When to choose one or another?
It seems that Embedding vector is the best solution here.
However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick".
In this variant, when the number of unique words ...
- 296
6
votes
Accepted
Do categorical features always need to be encoded?
You have partly answered this question yourself ("because converting to integers implies that there is an ordering between features").
I will just clarify the terminology a bit more.
Categorical ...
- 1,978
5
votes
Encoding features like month and hour as categorial or numeric?
It depends on which algorithm you're using.
If you're using tree-based algorithms like random forest, just pass this question. Categorical encoding isn't necessary for tree-based algorithms.
For ...
- 4,266
5
votes
Accepted
One hot encoding at character level with Keras
I think that you are looking for the keras Tokenizer with the char_level=True flag:
...
- 1,083
5
votes
Accepted
Muti-hot encoding vs Label-Encoding
You can think of binary encoding as a compromise between label encoding and one-hot encoding. For distinct categories, label encoding introduces a false linear order that brings a lot of noise into ...
oW_♦
- 6,254
5
votes
How do I encode the categorical columns if there are more than 15 unique values?
If you have high cardinality categorical data(+10 distinct values) you can do Target Encoding.
One hot Encoding in high cardinality scenarios has the following drawbacks:
The input data for the model ...
- 6,032
5
votes
How do I encode the categorical columns if there are more than 15 unique values?
While most answers here suggest to use various encoding schemes, I would like to propose a different approach: collapsing categories. The idea is that if there are two (or more) similar categories, ...
- 91
5
votes
What is the difference between one-hot and dummy encoding?
To complete Archana David's answer:
From what I encountered, the big advantage of sklearn.preprocessing.OneHotEncoder is that you can save it as an scikit-learn ...
- 854
4
votes
Accepted
Is it effective to use one-hot encoding when its dimension is as large as thousands
In neural networks applied to natural language processing normally each possible word (or sub-words) is handled as an individual token, and normally the vocabulary (the set of supported tokens) is ...
- 22.3k
4
votes
Always drop the first column after performing One Hot Encoding?
This question, in a slightly different form, was discussed herein earlier.
You are kind of right, but the best and safest way is to do One-Hot-Encoding and drop at the end because which column you ...
- 4,187
Only top scored, non community-wiki answers of a minimum length are eligible