27
votes
Accepted
Why is the decoder not a part of BERT architecture?
The need for an encoder depends on what your predictions are conditioned on, e.g.:
In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given ...
- 23.8k
20
votes
Accepted
What is the bleu score of professional human translators?
The original paper "BLEU: a Method for Automatic Evaluation of Machine Translation" contains a couple of numbers on this:
The BLEU metric ranges from 0 to 1. Few translations will attain a score of ...
- 5,360
12
votes
What's an LSTM-LM formulation?
The definition of a Language Model (LM) is a probability distribution over sequences of words.
The simple illustration of an LM is predicting the next word given the previous word(s).
For example, ...
- 2,186
9
votes
Multi-Head attention mechanism in transformer and need of feed forward neural network
The basic reasoning, I think, is just to increase capacity. While it is possible in theory for a single head, using multiple simply makes it easier. More specifically though, the paper says (pg 4):
...
- 1,010
8
votes
Accepted
What is the BLEU score used in Google Brain's "Attention Is All You Need" paper?
BLEU (Bi Lingual Evaluation Understudy) is an algorithm for evaluating the quality of text which has been machine-translated (MT) from one natural language to another. BLEU is typically measured on a ...
- 20.5k
7
votes
Accepted
What is context window size?
A context window applies to the number of words you will use to determine the context of each word. Like of your statement is "the quick brown fox" a context window of two would mean your samples are ...
- 1,848
6
votes
BPE vs WordPiece Tokenization - when to use / which?
(This answer was originally a comment)
You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end ...
- 23.8k
5
votes
What is the bleu score of professional human translators?
BLEU scores are based on comparing the translation to evaluate against a gold-standard translation. In general the gold-standard translation is the same source sentence translated by a professional ...
- 25k
4
votes
BPE vs WordPiece Tokenization - when to use / which?
Adding more info to noe's answer:
The difference between BPE and WordPiece lies in the way the symbol pairs are chosen for adding to the vocabulary. Instead of relying on the frequency of the pairs, ...
- 141
4
votes
Accepted
Why are there no automated translated subtitles?
OpenAI's Whisper is a widely known automatic speech recognition model that can also translate the input audio to text in the specified target language.
There are C++ implementations that can be used ...
- 23.8k
3
votes
Why is the decoder not a part of BERT architecture?
BERT is a pretraining model to do the downstream tasks such as question answering, NLI and other language tasks. So it just needs to encode the language representations so that it could be used for ...
3
votes
How to create a language translator from scratch?
Google translate itself uses Deep learning to translate sentences which can be seen here.
You can translate sentences across languages for which you need two things :
A large dataset which has pairs ...
- 2,180
3
votes
High Level Understanding of Neural Machine Translation
In encoder-decoder architecture, we first represent the input sequence by a fixed vector. It is assumed that this fix vector represents the complete meaning of the sentence. Now decoder uses this fix ...
- 81
3
votes
Language translation with convolutional neural network
I just googled:
A Convolutional Encoder Model for Neural Machine Translation, by Gehring et al., link
Convolutional Sequence to Sequence Learning, by Gehring et al. link
Pervasive Attention: 2D ...
- 6,095
3
votes
Transformer seq2seq model and loading embeddings from XLM-RoBERTa
It is indeed possible, but the question is if it is a good idea.
FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and ...
- 1,691
3
votes
Accepted
Reason for adding 1 to word index for sequence modeling
First, note that they are just adding 1 to the size of the vocabulary, not to the token IDs themselves, so the predictions are not affected.
Then, why adding 1 ?
Because ...
- 23.8k
3
votes
A good way to organize/store a lot of datasets
At this scale, grep is fast enough, so it's just a matter of having the files in a shape that's convenient for the most common types of searches.
At ModelFront, we ...
2
votes
Number of parameters in seq2seq model
I think I have found the answer, but I'd like to have some validation from the community. Could someone please let me know if this seems like a valid explanation?
The reason for the remaining 320M ...
- 31
2
votes
Why does LSTM performs better when the source target is reversed? (Seq2seq)
Also the paper said, we do not have a complete explanation to this phenomenon.
But here is kinda explanation about that.
While we do not have a complete explanation to this phenomenon, we
...
- 21
2
votes
Is there a dataset with news articles and their headlines?
The most widely used ones in text summarization research is the DUC dataset. If you see a paper using dataset "DUC 2015" or "DUC 2016" that's from here.
I have also personally used the Reuters ...
- 2,214
2
votes
Accepted
Do we really need <unk> tokens?
The <unk> tags can simply be used to tell the model that there is stuff, which is not semantically important to the output. This is a choice made via the ...
- 14.7k
2
votes
Is there "Attention Is All You Need" implementation in Keras?
Update for anyone googling this in 2021: Keras has implemented a MultiHead attention layer. If key, query, and value are the same, this is self-attention.
- 121
2
votes
Accepted
Is there "Attention Is All You Need" implementation in Keras?
Here is an implementation from PyPI.
- 136
2
votes
Problem with keras model loading
There seems to be an issue in Keras save_weights and load_model functions.
You can read more about it here, here and here. So far there is no workaround about it.
- 155
2
votes
How to build a machine translation system for a new language
I think your case would benefit from tunning a existing language to the new one but that is only a good approach if you plan to use it commercially. Also Google accepts help to improve their ...
- 1,626
2
votes
Accepted
How can I feed BERT to neural machine translation?
I find this way of using BERT in my translation system and it allows me to load and use more data to train my model.
I got a memory error when I want to use more data like 100k for my task. and I came ...
- 41
2
votes
Why is hard for neural machine translation model to learn rare words?
Rare words are not a problem only for NMT, they are a problem for MT in general. The reason is simple: in order to accurately translate a word in any particular context, the model needs to see as many ...
- 25k
2
votes
Why is the decoder not a part of BERT architecture?
I only understood tranformer decoders and encoders after I watched this video. Check it out:
https://www.youtube.com/watch?v=0_4KEb08xrE
- 121
2
votes
Sentences language translation with neural network, with a simple layer structure (if possible sequential)
Machine translation using traditional neural architecture (seq2seq models) had various issues due to rare-words, low accuracy and slow translation [1]. Even after using various mechanisms like ...
- 352
2
votes
Extract the embedding from a specific layer of MarianModel
You are passing discrete tokens as input to an attention layer that expects vectors of real numbers as inputs. The error you are getting tells you that the layer does not expect an input parameter ...
- 23.8k
Only top scored, non community-wiki answers of a minimum length are eligible