7

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM. A normal LM takes an autoregressive factorization of the probability of the sentence: $p(s) = \prod_t P(w_t | w_{<t})$ On the other hand, BERT's masked LM loss focuses on ...


6

The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not possible for the model to index the positional embedding for positions greater than the maximum. This limitation, nevertheless, is not arbitrary, but has a deeper ...


5

I will take as reference fairseq's implementation of the Transformer model. With this assumption: In the transformer, masks are used for two purposes: Padding: in the multi-head attention, the padding tokens are explicitly ignored by masking them. This corresponds to parameter key_padding_mask. Self-attention causality: in the multi-head attention blocks ...


4

The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it. The amount of memory needed by the self-attention in the Transformer is quadratic on the length of the input. This means that increasing the maximum length of the input, increases drastically the needed memory for self-...


4

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values). If you have many input classes, label smoothing can help. You can try a smaller model dimension. If you ...


3

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called word-pieces. On one hand, it ensures there are no out-of-vocabulary tokens, on the other hand, totally unknown words get split into characters and BERT probably ...


3

I'm not quite sure how's the decoder output is flattened into a single vector That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the sequence individually. These vectors have a fixed dimension, which is why it works.


3

pip install transformers Then try this from transformers import pipeline nlp = pipeline("fill-mask", model="bert-base") nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")


3

It is indeed possible, but the question is if it is a good idea. FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and replace the encoder with XLM-R. Another option would be using Huggingface's Transofrmers that also provides basic support for sequence-to-sequence models as ...


3

First let's understand why the format is like this. BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not. The [SEP] in the middle is here to help the model understand which token belong to which sentence. At ...


3

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the encoding. Here is the PE lookup table generation rewritten in C as a for-for loop: int d_model = 512, max_len = 5000; double pe[max_len][d_model]; for (int i = ...


2

A quick experiment you can do is to once do the preprocessing steps that you usually do and then feed it to the model and get the results. And once feed the dataset as it is to the model to compare the difference. In my experience doing the preprocessing won't make any difference, based on the dataset it gave me 1 more or less percent difference in accuracy (...


2

Its all ok. Exactly because your values are between 1 and 2 does he chooses 1.5. In other words trees will select 1 and 2 as values (integers not real numbers) but 1.5 is there as cut-off point between these classes.


2

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input. In order not to use a sequence of integers (1, 2, 3, ... n) because of the lack of boundary in the value and the magnitude, a float friendly is preferred. But, just using a limited (0 to 1) option means you ...


2

To include this logic into a pipeline you have to create a custom transformer. You need to ask yourself: [INIT] Are there any parameters in my logic? The variable you want to impute and the category you want this imputation to be based on. [FIT] What part of the logic is related to computing what the transformation will be? When you compute the median()...


2

class StrayCommaRemover(TransformerMixin): def __init__(self): //Initialize self by setting some variables here which can be passed as a input to transformer def fit(self, X, y=None): self.columns = X.columns //Setting context based on input Data return self def transform(self, X, y=None): // Actual transformation logic X= X.withColumn('...


2

So, the question talks about how to treat transformation choices as hyper parameters. How I would go about it is the following: Use one baseline model architecture for the data and then repeat the following: Instantiate the baseline model (effectively make sure all of the weights are initialised) Create the transformed dataset Train the model Compute ...


1

It's not a bug, although they added some confusion with this trick. They should better call their argument $j$ instead of $i$, cos what they actually do is they take all values $0 \leq j \leq d_{model} - 1$ and compute $PE(pos, j)$. $j$ сan be either even or odd, but in the right side of the equation it even, that's why they compute i//2 and multiply back by ...


1

So the question is concerned about understanding the self-attention mechanism in greater detail, in particular how this idea of multi-head self-attention is used to compute strength of relations between tokens. I think it's best you look through this great tutorial on self-attention and see if this helps in your understanding of multi-head self-attention: ...


1

They are meant for different purposes and they are hardly comparable. RoBERTa is meant for text classification and tagging tasks. The idea is that you take a pretrained RoBERTa model and finetune it on your (potentially small) classification or tagging dataset. Some examples of tasks where RoBERTa is useful are sentiment classification, part-of-speech (POS) ...


1

So the question asks between the difference between an attention vector and a positional vector. To answer this question, will give some context into how the transformer differs from a sequential model, such as RNNs and LSTMs. In the case of RNNs and LSTMs, data is fed sequentially "one-by-one" into the model to predict the output (whether that is ...


1

First to equation (7): $s_{i-1}$ is a vector, not a matrix. When you multiply it with a matrix $W$, you get another vector of which as the length of the intermediate attention projection, let us call it $d_a$. The shape of $W$ is thus $1024 \times d_a$. Similarly, the shape of $V$ is $512 \times d_a$ and bias $b$ is a vector of length $d_a$. The vector $w$ ...


1

Tensorflow handles batches differently on distribution strategies if you're using Keras, Estimator, or custom training loops. Since you are using TF1.15 Estimator with MirroredStrategy in one worker (1 machine), each replica (one per GPU) will receive a batch size of FLAGS.train_batch_size. So, if you have 4 GPUs, then the global batch size will be 4 * ...


1

The description says- ".................the features generated by each transformer will be concatenated to form a single feature space" Based on this I would not expect it to "reduce" the number of columns. On top of my mind, another pipeline which computes on dates column and feeds its output to numeric column transformation in columnTransformation.


1

I think your interpretation does not focus on the right places. First, let's understand where the predecessor idea of Transformer XL in LSTM Language Models: Truncated Back-Propagation Through Time (TBPTT). In LSTM Language Models, the training data text is concatenated into a single very long sequence. From this sequence, we create training batches in a ...


1

Firstly, fixing the input size of a model is more of an architectural decision than a problem. By fixing input size we: Limit the GPU memory usage while training Reduce the training time per epoch Reduce evaluation time per input sample If you want to, you can train your own transformer model by increasing the size of the input or increasing the number of ...


1

The general formulation of attention with queries, keys and values corresponds to a re-retrieval view on attention: you have some queries that you use to retrieve some values based on keys that correspond to them. With RNNs, attention is used for sequence-to-sequence models like machine translation. (Time series forecasting is usually formulated as sequence ...


1

Q, K, V vectors are trained with standard backpropagation. All trainable parameters are initialized at random, and then adjusted step by step with a Gradient Descent algorithm. Surprisingly, they are trained just as any standard ANN! It's pretty amazing what they can achieve with such a classical trick.


1

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous layers. In encoder-decoder attention, the queries are decoder states from the previous layer, keys and values and the encoder states. In Equation 1 of the ...


1

Let's take the common translation task which transformers can be used for as an example: If you would like to translate English to German one example of your training data could be ("the cat is black", "die Katze ist schwarz"). In this case your target is simply the German sentence "die Katze ist schwarz" (which is of course not processed as a string but ...


Only top voted, non community-wiki answers of a minimum length are eligible