68 votes

What is the positional encoding in the transformer model?

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be $$...
Esmailian's user avatar
  • 9,312
67 votes
Accepted

Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)

Use sparse categorical crossentropy when your classes are mutually exclusive (e.g. when each sample belongs exactly to one class) and categorical crossentropy when one sample can have multiple classes ...
featuredpeow's user avatar
60 votes
Accepted

What is the positional encoding in the transformer model?

Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations: Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings Taking ...
Batool's user avatar
  • 716
49 votes

Difference between OrdinalEncoder and LabelEncoder

Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while ...
ipramusinto's user avatar
38 votes

Sparse_categorical_crossentropy vs categorical_crossentropy (keras, accuracy)

The answer, in a nutshell If your targets are one-hot encoded, use categorical_crossentropy. Examples of one-hot encodings: ...
user78035's user avatar
  • 381
34 votes
Accepted

Encoding features like month and hour as categorial or numeric?

Have you considered adding the (sine, cosine) transformation of the time of day variable? This will ensure that the 0 and 23 hour for example are close to each other, thus allowing the cyclical nature ...
Pablo O's user avatar
  • 476
28 votes
Accepted

One Hot Encoding vs Word Embedding - When to choose one or another?

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each ...
Brian Spiering's user avatar
28 votes

Difference between OrdinalEncoder and LabelEncoder

As for differences in OrdinalEncoder and LabelEncoder implementation, the accepted answer mentions the shape of the data: ...
Nate Anderson's user avatar
19 votes
Accepted

How to deal with string labels in multi-class classification with keras?

Sklearn's LabelEncoder module finds all classes and assigns each a numeric id starting from 0. This means that whatever your class representations are in the ...
Neil Slater's user avatar
  • 28.9k
18 votes

Encoding features like month and hour as categorial or numeric?

The answer depends on the kind of relationships that you want to represent between the time feature, and the target variable. If you encode time as numeric, then you are imposing certain restrictions ...
raghu's user avatar
  • 641
16 votes
Accepted

Encoding with OrdinalEncoder : how to give levels as user input?

I'm not sure if you ever figured this out but I was trying to find answers on this exact same question and there aren't really any good answers in my opinion. I finally figured it out though. ...
fugumagu's user avatar
  • 176
14 votes
Accepted

One hot encoding alternatives for large categorical values

One option is to map rare values to 'other'. This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power. I have also ...
tom's user avatar
  • 2,248
14 votes

What is the positional encoding in the transformer model?

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take ...
Juan Esteban de la Calle's user avatar
12 votes
Accepted

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have ...
noe's user avatar
  • 26.6k
11 votes
Accepted

What is the difference between one-hot and dummy encoding?

Most machine learning models accept only numerical variables. This is the reason behind why categorical variables are converted to number so the model can understand better. Now lets address your ...
Archana David's user avatar
9 votes

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

the first few bits of the embedding are completely unusable by the network because the position encoding will distort them a lot This confused me very much at first because I was thinking of the ...
Denziloe's user avatar
  • 199
8 votes

Encoding features like month and hour as categorial or numeric?

I recommend using numerical features. Using categorical features essentially means that you don't consider distance between two categories as relevant (e.g. category 1 is as close to category 2 as it ...
Tanguy Coatalem's user avatar
8 votes
Accepted

How to handle columns with categorical data and many unique values

For categorical columns, you have two options : Entity Embeddings One Hot Vector For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change ...
Shamit Verma's user avatar
  • 2,249
8 votes

What is the positional encoding in the transformer model?

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think). They did not come up with the encoding. Here is the PE lookup table generation ...
Eris's user avatar
  • 81
8 votes

Why does frequency encoding work?

Check this post. In the cases where the frequency is related somewhat with the target variable, it helps the model to understand and assign the weight in direct and inverse proportion, depending on ...
Piotr Rarus's user avatar
8 votes

What is the difference between one-hot and dummy encoding?

The purpose of one-hot encoding is to assign numbers to categorical variables which does not create a false, meaningless numerical pattern. If you have categorical variables "Apple", "...
Misha Lavrov's user avatar
7 votes
Accepted

Pandas categorical variables encoding for regression (one-hot encoding vs dummy encoding)

One advantage of get_dummies is that it can operate on values other than integers (so you don't need the LabelEncoder) and ...
oW_'s user avatar
  • 6,347
6 votes

One Hot Encoding vs Word Embedding - When to choose one or another?

It seems that Embedding vector is the best solution here. However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick". In this variant, when the number of unique words ...
michaelg's user avatar
  • 296
6 votes
Accepted

Do categorical features always need to be encoded?

You have partly answered this question yourself ("because converting to integers implies that there is an ordering between features"). I will just clarify the terminology a bit more. Categorical ...
hssay's user avatar
  • 1,998
6 votes

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

The confusion here is that we believe positional embedding is a more complicated version of adding positional information to the word embedding; however, it is not actually. Adding new dimensions to ...
Hamid Mohammadi's user avatar
6 votes

What is the difference between one-hot and dummy encoding?

To complete Archana David's answer: From what I encountered, the big advantage of sklearn.preprocessing.OneHotEncoder is that you can save it as an scikit-learn ...
Adept's user avatar
  • 864
5 votes

Encoding features like month and hour as categorial or numeric?

It depends on which algorithm you're using. If you're using tree-based algorithms like random forest, just pass this question. Categorical encoding isn't necessary for tree-based algorithms. For ...
Icyblade's user avatar
  • 4,326
5 votes
Accepted

One hot encoding at character level with Keras

I think that you are looking for the keras Tokenizer with the char_level=True flag: ...
Adrien D's user avatar
  • 1,113
5 votes
Accepted

Muti-hot encoding vs Label-Encoding

You can think of binary encoding as a compromise between label encoding and one-hot encoding. For distinct categories, label encoding introduces a false linear order that brings a lot of noise into ...
oW_'s user avatar
  • 6,347
5 votes
Accepted

Applying mean encoding before or after splitting into train and test set

The purpose of having a test set or a validation set is to be able to check the performance of your model on data it has not seen before. If you perform feature engineering with the test data present ...
Simon Larsson's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible