Skip to main content
31 votes
Accepted

Why is the decoder not a part of BERT architecture?

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given ...
noe's user avatar
  • 27k
20 votes
Accepted

What is the bleu score of professional human translators?

The original paper "BLEU: a Method for Automatic Evaluation of Machine Translation" contains a couple of numbers on this: The BLEU metric ranges from 0 to 1. Few translations will attain a score of ...
Jonathan's user avatar
  • 5,430
12 votes

What's an LSTM-LM formulation?

The definition of a Language Model (LM) is a probability distribution over sequences of words. The simple illustration of an LM is predicting the next word given the previous word(s). For example, ...
Rizky Luthfianto's user avatar
9 votes

Multi-Head attention mechanism in transformer and need of feed forward neural network

The basic reasoning, I think, is just to increase capacity. While it is possible in theory for a single head, using multiple simply makes it easier. More specifically though, the paper says (pg 4): ...
user3658307's user avatar
  • 1,020
8 votes
Accepted

What is the BLEU score used in Google Brain's "Attention Is All You Need" paper?

BLEU (Bi Lingual Evaluation Understudy) is an algorithm for evaluating the quality of text which has been machine-translated (MT) from one natural language to another. BLEU is typically measured on a ...
Brian Spiering's user avatar
6 votes
Accepted

What is context window size?

A context window applies to the number of words you will use to determine the context of each word. Like of your statement is "the quick brown fox" a context window of two would mean your samples are ...
Himanshu Rai's user avatar
  • 1,848
6 votes

BPE vs WordPiece Tokenization - when to use / which?

(This answer was originally a comment) You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end ...
noe's user avatar
  • 27k
5 votes

Why is the decoder not a part of BERT architecture?

BERT is a pretraining model to do the downstream tasks such as question answering, NLI and other language tasks. So it just needs to encode the language representations so that it could be used for ...
vishwajit kumar vishnu's user avatar
5 votes

What is the bleu score of professional human translators?

BLEU scores are based on comparing the translation to evaluate against a gold-standard translation. In general the gold-standard translation is the same source sentence translated by a professional ...
Erwan's user avatar
  • 25.5k
4 votes

Why is the decoder not a part of BERT architecture?

In short, Bidirectional Encoder Representations from Transformers (BERT) is not designed for decorder-related tasks. I can't see how BERT makes predictions without using a decoder unit, which was a ...
We.Me.ii's user avatar
4 votes

BPE vs WordPiece Tokenization - when to use / which?

Adding more info to noe's answer: The difference between BPE and WordPiece lies in the way the symbol pairs are chosen for adding to the vocabulary. Instead of relying on the frequency of the pairs, ...
Abhi25t's user avatar
  • 141
4 votes
Accepted

Why are there no automated translated subtitles?

OpenAI's Whisper is a widely known automatic speech recognition model that can also translate the input audio to text in the specified target language. There are C++ implementations that can be used ...
noe's user avatar
  • 27k
3 votes

Why is the decoder not a part of BERT architecture?

First need to understand what problems BERT can solve or what kind of inference/prediction it can achieve. BERT Neural Network - EXPLAINED! Encoder in Transformer itself can learn: Relations among ...
mon's user avatar
  • 731
3 votes

How to create a language translator from scratch?

Google translate itself uses Deep learning to translate sentences which can be seen here. You can translate sentences across languages for which you need two things : A large dataset which has pairs ...
Shubham Panchal's user avatar
3 votes

High Level Understanding of Neural Machine Translation

In encoder-decoder architecture, we first represent the input sequence by a fixed vector. It is assumed that this fix vector represents the complete meaning of the sentence. Now decoder uses this fix ...
tourism's user avatar
  • 81
3 votes

Language translation with convolutional neural network

I just googled: A Convolutional Encoder Model for Neural Machine Translation, by Gehring et al., link Convolutional Sequence to Sequence Learning, by Gehring et al. link Pervasive Attention: 2D ...
Leevo's user avatar
  • 6,265
3 votes

Transformer seq2seq model and loading embeddings from XLM-RoBERTa

It is indeed possible, but the question is if it is a good idea. FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and ...
Jindřich's user avatar
  • 1,751
3 votes
Accepted

Reason for adding 1 to word index for sequence modeling

First, note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves, so the predictions are not affected. Then, why adding 1 ? Because ...
noe's user avatar
  • 27k
3 votes

A good way to organize/store a lot of datasets

At this scale, grep is fast enough, so it's just a matter of having the files in a shape that's convenient for the most common types of searches. At ModelFront, we ...
Adam Bittlingmayer's user avatar
2 votes

Number of parameters in seq2seq model

I think I have found the answer, but I'd like to have some validation from the community. Could someone please let me know if this seems like a valid explanation? The reason for the remaining 320M ...
Sharkovsky87's user avatar
2 votes

Why does LSTM performs better when the source target is reversed? (Seq2seq)

Also the paper said, we do not have a complete explanation to this phenomenon. But here is kinda explanation about that. While we do not have a complete explanation to this phenomenon, we ...
korchris's user avatar
2 votes

Is there a dataset with news articles and their headlines?

The most widely used ones in text summarization research is the DUC dataset. If you see a paper using dataset "DUC 2015" or "DUC 2016" that's from here. I have also personally used the Reuters ...
user12075's user avatar
  • 2,284
2 votes
Accepted

Do we really need <unk> tokens?

The <unk> tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the ...
n1k31t4's user avatar
  • 14.9k
2 votes

Is there "Attention Is All You Need" implementation in Keras?

Update for anyone googling this in 2021: Keras has implemented a MultiHead attention layer. If key, query, and value are the same, this is self-attention.
Tanner Phillips's user avatar
2 votes
Accepted

Is there "Attention Is All You Need" implementation in Keras?

Here is an implementation from PyPI.
eugen's user avatar
  • 136
2 votes

Problem with keras model loading

There seems to be an issue in Keras save_weights and load_model functions. You can read more about it here, here and here. So far there is no workaround about it.
agcala's user avatar
  • 155
2 votes

How to build a machine translation system for a new language

I think your case would benefit from tunning a existing language to the new one but that is only a good approach if you plan to use it commercially. Also Google accepts help to improve their ...
Pedro Henrique Monforte's user avatar
2 votes
Accepted

How can I feed BERT to neural machine translation?

I find this way of using BERT in my translation system and it allows me to load and use more data to train my model. I got a memory error when I want to use more data like 100k for my task. and I came ...
Hamed's user avatar
  • 39
2 votes

What is Bit Per Character?

Perplexity is a measurement of how well a probability model predicts a sample. Intuitively, perplexity can be understood as a measure of uncertainty. Say the real thing you want to predict is the ...
Pluviophile's user avatar
  • 3,918
2 votes

Why is hard for neural machine translation model to learn rare words?

Rare words are not a problem only for NMT, they are a problem for MT in general. The reason is simple: in order to accurately translate a word in any particular context, the model needs to see as many ...
Erwan's user avatar
  • 25.5k

Only top scored, non community-wiki answers of a minimum length are eligible