Skip to main content
5 votes
Accepted

What is the difference between BERT architecture and vanilla Transformer architecture

The name provides a clue. BERT (Bidirectional Encoder Representations from Transformers): So basically BERT = Transformer Minus the Decoder BERT ends with the final representation of the words after ...
Allohvk's user avatar
  • 888
4 votes
Accepted

Why transform embedding dimension in sin-cos positional encoding?

First, let's reason why positional embeddings are needed at all: A multi-head attention layer of the Transformer architecture performs computations that are position-independent. This means that, if ...
noe's user avatar
  • 27k
4 votes
Accepted

What to do with Transformer Encoder output?

The typical approach for this is follow BERT's approach: add an extra special token at the beginning of the input sequence (in BERT it is [CLS]) and only use the ...
noe's user avatar
  • 27k
2 votes
Accepted

SkLearn Categorical Naive Bayes Vs Mathematical theory of Naive Bayes

Now if we get a new data (Age = young, Income= fair) we need to find out in which class this data should belong. ... example 1) If a sample doesn't have a label you can't include it in the train/test ...
Kenneth Granahan's user avatar
2 votes
Accepted

sklearn serialize label encoder for multiple categorical columns

LabelEncoder is meant for the labels (target, dependent variable), not for the features. OrdinalEncoder can be used for ...
Ben Reiniger's user avatar
  • 11.9k
1 vote

Encoding correlation

You could do that with a neural autoencoder using a custom loss function. Use a hidden layer, let's call it $l_{encoded}$, with more nodes than the features of the input data. You have to code the ...
PascalIv's user avatar
  • 432
1 vote

How is RNN decoder output calculated?

The paper's Appendix A has the exact formulas used. Specifically A.1 contains the exact formula for calculating $\mathbf{c}$, and A.1.1 contains details on where and how it is used.
masaers's user avatar
  • 141
1 vote
Accepted

What does the output of an encoder in encoder-decoder model represent?

Encoder-decoder with RNNs With RNNs, you can either use the hidden state of the encoder's last time step (i.e. return_sequences=False in Keras) or use the outputs/...
noe's user avatar
  • 27k
1 vote

What does the output of an encoder in encoder-decoder model represent?

If you're using an RNN architecture as your encoder, say an LSTM or GRU, then the output of your encoder is the hidden-state representation of each time step in your input. So for each example that ...
AleksJ's user avatar
  • 41
1 vote

Encode categorical data for unsupervised learning

Different unsupervised machine learning algorithms have different assumptions. K-means clustering requires computing euclidean distance thus all encodings have to be consistent with euclidean distance ...
Brian Spiering's user avatar
1 vote

Is it vital to do label encoding with target variable

If by label encoding you mean one-hot-encoding, no it's not necessary. In fact it's not a good idea because this would create two target variables instead of one, a setting which corresponds to multi-...
Erwan's user avatar
  • 25.5k
1 vote

How to Visualize attention weights in a Attention based Encoder-Decoder network in Time series forecasting

A reference: https://github.com/zhaocq-nlp/Attention-Visualization/blob/master/exec/plot_heatmap.py. It plots attentions based on matplotlib
Rancho Xia's user avatar
1 vote
Accepted

Changing order of LabelEncoder() result

Do the labels have to start from 0? No it doesn't matter where they start as long as they have distinct values. Do the labels have to be sequential? Well it depends from the feature. For example if ...
Giannis Krilis's user avatar
1 vote

Encode time-series of different lengths with keras

Figure out the max length you want your time series to be and use linear interpolation to fill in missing values in the shorter series so that time series can be same length. Example of this is: ...
Z. Cass's user avatar
  • 11
1 vote

Role of decoder in Transformer?

Further details regarding your last question. Your problem is a sequence classification exercise. Decoders aren't needed. You can use a Dense layer to predict the labels. You can even add 'Attention'-...
Allohvk's user avatar
  • 888
1 vote
Accepted

Role of decoder in Transformer?

Yes, you are right in your understanding of the role of the decoder. However, your use of "query" here, while somewhat technically correct, seems a bit strange. You are referring as "...
noe's user avatar
  • 27k

Only top scored, non community-wiki answers of a minimum length are eligible