New answers tagged transformer
0
The input dimensionality is the embedding size, which is the same as the model dimensionality, as explained in section 3.4 of the article:
3.4 Embeddings and Softmax
Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension $d_{model}$.
Therefore, the input ...
0
The last step for production models is typically to train with the entire set (train + validation) after tuning the hyperparameters using the validation set(s). The difference is typically not too drastic as the validation set should only be a fraction of the dataset but more data is always helpful especially for DL-based models.
I'm not familiar with the ...
2
There is at least one way:
Create/Acquire a grammar model for the language spoken (there are several such models for various languages used in NLP)
Test the transcripts for beign grammaticaly/syntacticaly correct.
This assesment will at least rule out gibberish and most of transcripts that do not correspond to valid sentences of the language spoken
1
Your understanding is not correct.
The relevant information is described in the original paper in section 3.2.2:
The three sets of projection matrices you are referring to are $W^Q_i \in \mathbb{R}^{d_{model} \times d_k}$ for the Queries, $W^K_i \in \mathbb{R}^{d_{model}\times d_k}$ for the Keys and $W^V_i \in \mathbb{R}^{d_{model}\times d_v}$ for the Values....
0
Thank-you!! I'd also missed that multiply in my (fairseq transformer) code study, and it helps clear up a mystery that I'd noted: the (sinusoidal, non-learned) positional embeddings are initialized with a range of -1.0 to +1.0, but the word-embeddings are initialized with a mean of 0.0 and s.d. of embedding_dim ** -0.5 (0.044 for 512, 0.03125 for 1024).
So, ...
1
If the masking were only applied in the first layer, the self-attention in the subsequent layers would bring to each position information from future tokens.
Let's break it down with numbers:
At layer $i$, if causal masking is applied, the output at position $t$ contains information about layer $i-1$ at positions $1..t-1$, that is, $L_{i,t} = f_i(L_{i-1,1},....
0
According to BERT-GPU.
No problem with the code.
The batch_size is for 1 GPU.
0
If the labeling people is sufficient,
the best solution is to re-label the noisy data.
0
@Jindřich is exactly right. this is not an answer by itself, but rather some pointers to the implementation in the annotated transformer of every item he mentioned:
The input of the dense layers is of shape $b \times l \times d_m$:
torch.from_numpy(np.random.randint(1, V, size=(batch, l))): $b \times l$
self.lut = nn.Embedding(vocab, d_model); self.lut(x): ...
6
This is specified in the original Transformer paper, at the end of section 3.4:
Transcription:
3.4 Embeddings and Softmax
Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension 𝑑model. We also use the usual learned linear transformation and softmax function to ...
1
BERT is a Transformer encoder, while GPT is a Transformer decoder:
You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the masking in the multi-head attention block.
There is, however, an extra difference in how BERT and GPT are trained:
BERT is a Transformer ...
1
It is not very clear what you are referring to with "number of input neurons".
The input layer in BERT is an embedding layer, which is a table of vectors. Each of those vectors has dimensionality 768, and each vector is associated to one of the tokens in the vocabulary (so the number of vectors in the embedding table is the vocabulary size).
0
Removing stop words or keeping them is an empirical question. The effect will vary based on corpus and task. In fact, the definition of stop words depends on the corpus and the task.
One approach would be to benchmark the effect of stop words with cross validation for the specific scenario.
2
Very interesting question.
Easy, but probably lazy answer
When using pre-trained models, it is always advised to feed it data similar to what it was trained with. Basically, if it matters, don't remove them, and if it doesn't matter, it doesn't hurt to keep them in. Obviously, if you can, try with or without stopwords, and see what works best for your ...
0
All the tokens are inferred at once, independently from one another.
Top 50 recent answers are included
