# Tag Info

44

Use sparse categorical crossentropy when your classes are mutually exclusive (e.g. when each sample belongs exactly to one class) and categorical crossentropy when one sample can have multiple classes or labels are soft probabilities (like [0.5, 0.3, 0.2]). Formula for categorical crossentropy (S - samples, C - classess, $s \in c$ - sample belongs to class ...

38

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be \begin{align*}e_{w}' &= e_{w} + \left[sin\left(\frac{pos}{10000^{0}}\right), cos\left(\frac{pos}{10000^{0}}\right),sin\left(\frac{pos}{10000^{2/4}}\right),...

24

Have you considered adding the (sine, cosine) transformation of the time of day variable? This will ensure that the 0 and 23 hour for example are close to each other, thus allowing the cyclical nature of the variable to shine through. (More Info)

19

Afaik, both have the same functionality. A bit difference is the idea behind. OrdinalEncoder is for converting features, while LabelEncoder is for converting target variable. That's why OrdinalEncoder can fit data that has the shape of (n_samples, n_features) while LabelEncoder can only fit data that has the shape of (n_samples,) (though in the past one ...

18

Sklearn's LabelEncoder module finds all classes and assigns each a numeric id starting from 0. This means that whatever your class representations are in the original data set, you now have a simple consistent way to represent each. It doesn't do one-hot encoding, although as you correctly identify, it is pretty close, and you can use those ids to quickly ...

16

The Answer, In a Nutshell If your targets are one-hot encoded, use categorical_crossentropy. Examples of one-hot encodings: [1,0,0] [0,1,0] [0,0,1] But if your targets are integers, use sparse_categorical_crossentropy. Examples of integer encodings (for the sake of completion): 1 2 3

12

The answer depends on the kind of relationships that you want to represent between the time feature, and the target variable. If you encode time as numeric, then you are imposing certain restrictions on the model. For a linear regression model, the effect of time is now monotonic, either the target will increase or decrease with time. For decision trees, ...

11

One option is to map rare values to 'other'. This is commonly done in e.g. natural language processing - the intuition being that very rare labels don't carry much statistical power. I have also seen people map 1-hot categorical values to lower-dimensional vectors, where each 1-hot vector is re-represented as a draw from a multivariate Gaussian. See e.g. ...

8

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each new category. However, that speed and simplicity also leads to the "curse of dimensionality" by creating a new dimension for each category. Embedding is a ...

8

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take into account that sentences could be of any length, so saying '"X" word is the third in the sentence' does not make sense if there are different length sentences:...

7

I recommend using numerical features. Using categorical features essentially means that you don't consider distance between two categories as relevant (e.g. category 1 is as close to category 2 as it is to category 3). This is definitely not the case for hours or months. However, the issue that you raise is that you want to represent hours and months in a ...

7

One advantage of get_dummies is that it can operate on values other than integers (so you don't need the LabelEncoder) and returns a DataFrame with the categories as column names. Also, you can conveniently drop one redundant category using drop_first=True. One advantage of scikit-learn's OneHoteEncoder lies in the scikit-learn API. OHE gives you a ...

7

For categorical columns, you have two options : Entity Embeddings One Hot Vector For a column with 145 values, I would use one hot encoding and Embedding for ~3k values. This decision might change depending on overall number of features. Embeddings map feature values into a 1D vector so that model knows NYC, Paris, London are similar cities in one ...

6

You have partly answered this question yourself ("because converting to integers implies that there is an ordering between features"). I will just clarify the terminology a bit more. Categorical data: information has categories, but no natural ordering defined between them (gender, name of user's cat) Ordinal data: information has categories with natural ...

6

Check this post. In the cases where the frequency is related somewhat with the target variable, it helps the model to understand and assign the weight in direct and inverse proportion, depending on the nature of the data. Check also this thread. What's the rationale behind it? High cardinality may result in dimensionality curse and actually decrease ...

5

I think that you are looking for the keras Tokenizer with the char_level=True flag: from keras.preprocessing.text import Tokenizer tokenizer = Tokenizer(char_level=True) tokenizer.fit_on_texts(your_dataset_train) sequence_of_int = tokenizer.texts_to_sequences(your_dataset_train_or_test) Now that you have sequences of Integer, you can use keras.utils....

5

It seems that Embedding vector is the best solution here. However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick". In this variant, when the number of unique words is too large to be assigned a unique index in a dictionary, one may hash words of into vector of fixed size. One advantage in your use case is that you may ...

5

As for differences in OrdinalEncoder and LabelEncoder implementation, the accepted answer mentions the shape of the data: (OrdinalEncoder for 2D data; shape (n_samples, n_features), LabelEncoder is for 1D data: for shape (n_samples,)) That's why a OrdinalEncoder would get an error: ValueError: Expected 2D array, got 1D array instead: ...if trying to fit on ...

4

It depends on which algorithm you're using. If you're using tree-based algorithms like random forest, just pass this question. Categorical encoding isn't necessary for tree-based algorithms. For other algorithms like neural network, I suggest trying both method(continuous & categorical). The effect differs between different situations.

4

In neural networks applied to natural language processing normally each possible word (or sub-words) is handled as an individual token, and normally the vocabulary (the set of supported tokens) is around 32K. These tokens are typically one-hot encoded. Therefore, there is not inherent problem in having one-hot encoded vectors with thousands of components. ...

4

This question, in a slightly different form, was discussed herein earlier. You are kind of right, but the best and safest way is to do One-Hot-Encoding and drop at the end because which column you want to drop at the very beginning? In fact in pandas.get_dummies there is a parameter i.e. drop_first allows you whether to keep or remove the reference (...

3

Personally, I think that building a predictor on such user Ids is not that efficient neither useful. Can you change your problem in a different manner? When we do any analysis we drop those ids as they are of no use they are only the incidents in a large population and we try to group them in categories which describe their features. For example, users ...

3

Consider the following chunk of data: 1010010110100101 Universal - these are generic compression algorithms that are data agnostic. A crude version of run length encoding would fall into this category. The advantage is that it is very fast to compress and decompress. The downside is that it may be extremely ineffective based on the data to be compressed. ...

3

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the encoding. Here is the PE lookup table generation rewritten in C as a for-for loop: int d_model = 512, max_len = 5000; double pe[max_len][d_model]; for (int i = ...

3

The purpose of having a test set or a validation set is to be able to check the performance of your model on data it has not seen before. If you perform feature engineering with the test data present you will get a data leakage. That happens when give your model information about your test data during training. It is especially bad when doing target ...

3

This is actually a feature of tree-based models in general, not just gradient boosting trees. Not exactly a reference, but this Medium article explains why ordinal encoding is often more efficient. On the topic of safety, I think the author should have said that the use of ordinal encoding is more safe compared to linear methods, but still not perfectly safe....

2

LabelEncoder converts strings to integers, but you have integers already. Thus, LabelEncoder will not help you anyway. Wenn you are using your column with integers as it is, sklearn treats it as numbers. This means, for example, that distance between 1 and 2 is 1, distance between 1 and 4 is 3. Can you say the same about your activities (if you know the ...

2

You're doing something wrong. I can query a 100K word dict in nanoseconds word_list = open('/usr/share/dict/words').read().split() len(word_list) > 99171 word_dict = {word: hash(word) for word in word_list} %timeit word_dict['blazing'] > 10000000 loops, best of 3: 33.8 ns per loop

2

I'd like to extend the great @Emre's answer with another example - we are going to replace all tokenized words from the "1984" (c) George Orwell (120K words): In [163]: %paste import requests import nltk import pandas as pd # source: https://github.com/dwyl/english-words fn = r'D:\temp\.data\words.txt' url = 'http://gutenberg.net.au/ebooks01/0100021.txt' ...

2

You can read the data and first get a list of all the unique values of your categorical variables. Then you can fit a one hot encoder object (like the sklearn.preprocessing.CategoricalEncoder) on your list of unique values. This method can also help in a train test framework or when you are reading your data in chunks. I have created a python module that ...

Only top voted, non community-wiki answers of a minimum length are eligible