58

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be $$\begin{align*}e_{w}' &= e_{w} + \left[sin\left(\frac{pos}{10000^{0}}\right), cos\left(\frac{pos}{10000^{0}}\right),sin\left(\frac{pos}{10000^{2/4}}\right),...


55

Use sparse categorical crossentropy when your classes are mutually exclusive (e.g. when each sample belongs exactly to one class) and categorical crossentropy when one sample can have multiple classes or labels are soft probabilities (like [0.5, 0.3, 0.2]). Formula for categorical crossentropy (S - samples, C - classess, $s \in c $ - sample belongs to class ...


36

Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while LabelEncoder is for converting target variable. That's why OrdinalEncoder can fit data that has the shape of (n_samples, n_features) while LabelEncoder can only fit data that has the shape of (n_samples,) (though in the past one ...


32

Have you considered adding the (sine, cosine) transformation of the time of day variable? This will ensure that the 0 and 23 hour for example are close to each other, thus allowing the cyclical nature of the variable to shine through. (More Info)


29

The answer, in a nutshell If your targets are one-hot encoded, use categorical_crossentropy. Examples of one-hot encodings: [1,0,0] [0,1,0] [0,0,1] But if your targets are integers, use sparse_categorical_crossentropy. Examples of integer encodings (for the sake of completion): 1 2 3


19

Sklearn's LabelEncoder module finds all classes and assigns each a numeric id starting from 0. This means that whatever your class representations are in the original data set, you now have a simple consistent way to represent each. It doesn't do one-hot encoding, although as you correctly identify, it is pretty close, and you can use those ids to quickly ...


18

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each new category. However, that speed and simplicity also leads to the "curse of dimensionality" by creating a new dimension for each category. Embedding ...


17

The answer depends on the kind of relationships that you want to represent between the time feature, and the target variable. If you encode time as numeric, then you are imposing certain restrictions on the model. For a linear regression model, the effect of time is now monotonic, either the target will increase or decrease with time. For decision trees, ...


17

Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations: Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings Taking excerpts from the video, let us try understanding the “sin” part of the formula to compute the position embeddings: Here “pos” refers to the position of the “...


16

As for differences in OrdinalEncoder and LabelEncoder implementation, the accepted answer mentions the shape of the data: OrdinalEncoder is for 2D data with the shape (n_samples, n_features) LabelEncoder is for 1D data with the shape (n_samples,)) Maybe that's why the top-voted answer suggests OrdinalEncoder is for the "features" (often a 2D ...


12

One option is to map rare values to 'other'. This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power. I have also seen people map 1-hot categorical values to lower-dimensional vectors, where each 1-hot vector is re-represented as a draw from a multivariate Gaussian. See e.g. ...


12

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take into account that sentences could be of any length, so saying '"X" word is the third in the sentence' does not make sense if there are different length sentences:...


8

I recommend using numerical features. Using categorical features essentially means that you don't consider distance between two categories as relevant (e.g. category 1 is as close to category 2 as it is to category 3). This is definitely not the case for hours or months. However, the issue that you raise is that you want to represent hours and months in a ...


7

One advantage of get_dummies is that it can operate on values other than integers (so you don't need the LabelEncoder) and returns a DataFrame with the categories as column names. Also, you can conveniently drop one redundant category using drop_first=True. One advantage of scikit-learn's OneHoteEncoder lies in the scikit-learn API. OHE gives you a ...


7

For categorical columns, you have two options : Entity Embeddings One Hot Vector For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features. Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one ...


7

Check this post. In the cases where the frequency is related somewhat with the target variable, it helps the model to understand and assign the weight in direct and inverse proportion, depending on the nature of the data. Check also this thread. What's the rationale behind it? High cardinality may result in dimensionality curse and actually decrease ...


6

It seems that Embedding vector is the best solution here. However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick". In this variant, when the number of unique words is too large to be assigned a unique index in a dictionary, one may hash words of into vector of fixed size. One advantage in your use case is that you may ...


6

You have partly answered this question yourself ("because converting to integers implies that there is an ordering between features"). I will just clarify the terminology a bit more. Categorical data: information has categories, but no natural ordering defined between them (gender, name of user's cat) Ordinal data: information has categories with natural ...


6

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the encoding. Here is the PE lookup table generation rewritten in C as a for-for loop: int d_model = 512, max_len = 5000; double pe[max_len][d_model]; for (int i = ...


6

I'm not sure if you ever figured this out but I was trying to find answers on this exact same question and there aren't really any good answers in my opinion. I finally figured it out though. OrdinalEncoder is capable of encoding multiple columns in a dataframe. So, when you instantiate OrdinalEncoder(), you give the categories parameter a list of lists: enc ...


6

Most machine learning models accept only numerical variables. This is the reason behind why categorical variables are converted to number so the model can understand better. Now lets address your second query lets look into what is one-hot encoding and dummy encoding and then see the difference One hot Encoding: Take the example of column name Fruit which ...


5

It depends on which algorithm you're using. If you're using tree-based algorithms like random forest, just pass this question. Categorical encoding isn't necessary for tree-based algorithms. For other algorithms like neural network, I suggest trying both method(continuous & categorical). The effect differs between different situations.


5

I think that you are looking for the keras Tokenizer with the char_level=True flag: from keras.preprocessing.text import Tokenizer tokenizer = Tokenizer(char_level=True) tokenizer.fit_on_texts(your_dataset_train) sequence_of_int = tokenizer.texts_to_sequences(your_dataset_train_or_test) Now that you have sequences of Integer, you can use keras.utils....


5

You can think of binary encoding as a compromise between label encoding and one-hot encoding. For distinct categories, label encoding introduces a false linear order that brings a lot of noise into the model (category 1 < category 2 < category 3....) . Binary encoding introduces false additive relationships between the categories (e.g. category 4 + ...


4

In neural networks applied to natural language processing normally each possible word (or sub-words) is handled as an individual token, and normally the vocabulary (the set of supported tokens) is around 32K. These tokens are typically one-hot encoded. Therefore, there is not inherent problem in having one-hot encoded vectors with thousands of components. ...


4

This question, in a slightly different form, was discussed herein earlier. You are kind of right, but the best and safest way is to do One-Hot-Encoding and drop at the end because which column you want to drop at the very beginning? In fact in pandas.get_dummies there is a parameter i.e. drop_first allows you whether to keep or remove the reference (whether ...


4

The purpose of having a test set or a validation set is to be able to check the performance of your model on data it has not seen before. If you perform feature engineering with the test data present you will get a data leakage. That happens when give your model information about your test data during training. It is especially bad when doing target ...


4

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have to define two dimensionalities, $d_t$ for the token and $d_p$ for the position, with the total dimensionality $d = d_t + d_p$, so $d>d_t$ and $d>d_p$. We ...


4

First, I generally agree that encoding unordered categories as consecutive integers is not a great approach: you are adding a ton of additional relationships that aren't present in the data. CART First, let me point out (because I nearly forgot) that there are two main types of decision tree: CART and the Quinlan family. For the Quinlan family, categorical ...


4

If your algorithm is based on gradient descent optimization, you can use embeddings, which are dense representation spaces for discrete elements. Embeddings are supported by most deep learning frameworks such as pytorch or tensorflow. Update: the fact that you want to have multiple discrete values does not prevent the possibility of using embeddings: you can ...


Only top voted, non community-wiki answers of a minimum length are eligible