40

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be $$\begin{align*}e_{w}' &= e_{w} + \left[sin\left(\frac{pos}{10000^{0}}\right), cos\left(\frac{pos}{10000^{0}}\right),sin\left(\frac{pos}{10000^{2/4}}\right),...


15

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling. BERT is trained on a masked language modeling task and therefore you cannot "predict the next word". You can only mask a word and ask BERT to predict it given the rest of the sentence (both to the left and to the right of the masked ...


10

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take into account that sentences could be of any length, so saying '"X" word is the third in the sentence' does not make sense if there are different length sentences:...


9

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM. A normal LM takes an autoregressive factorization of the probability of the sentence: $p(s) = \prod_t P(w_t | w_{<t})$ On the other hand, BERT's masked LM loss focuses on ...


7

At each decoding time step, the decoder receives 2 inputs: the encoder output: this is computed once and is fed to all layers of the decoder at each decoding time step as key ($K_{endec}$) and value ($V_{endec}$) for the encoder-decoder attention blocks. the target tokens decoded up to the current decoding step: for the first step, the matrix contains in ...


6

The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not possible for the model to index the positional embedding for positions greater than the maximum. This limitation, nevertheless, is not arbitrary, but has a deeper ...


5

I will take as reference fairseq's implementation of the Transformer model. With this assumption: In the transformer, masks are used for two purposes: Padding: in the multi-head attention, the padding tokens are explicitly ignored by masking them. This corresponds to parameter key_padding_mask. Self-attention causality: in the multi-head attention blocks ...


4

The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it. The amount of memory needed by the self-attention in the Transformer is quadratic on the length of the input. This means that increasing the maximum length of the input, increases drastically the needed memory for self-...


4

pip install transformers Then try this from transformers import pipeline nlp = pipeline("fill-mask", model="bert-base") nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")


4

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values). If you have many input classes, label smoothing can help. You can try a smaller model dimension. If you ...


3

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the encoding. Here is the PE lookup table generation rewritten in C as a for-for loop: int d_model = 512, max_len = 5000; double pe[max_len][d_model]; for (int i = ...


3

First let's understand why the format is like this. BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not. The [SEP] in the middle is here to help the model understand which token belong to which sentence. At ...


3

It is indeed possible, but the question is if it is a good idea. FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and replace the encoder with XLM-R. Another option would be using Huggingface's Transofrmers that also provides basic support for sequence-to-sequence models as ...


3

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called word-pieces. On one hand, it ensures there are no out-of-vocabulary tokens, on the other hand, totally unknown words get split into characters and BERT probably ...


3

I'm not quite sure how's the decoder output is flattened into a single vector That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the sequence individually. These vectors have a fixed dimension, which is why it works.


2

According to [Caccia et al., 2018], in general textual GANs are no rival for LMs regarding several quality measures. These are the conclusions of the paper: This research demonstrates that well-adjusted language models are a remarkably strong baseline and that temperature sweeping can provide a very clear characterization of model performance. A well-...


2

Neural networks are black boxes and any answer would be pure speculation, especially not knowing what fine tuning task we are speaking about. Only by running an experiment would you get the actual answer, and not even then, given the loose definition of the formulations "integrate more information" and "change a lot" in your questions.


2

A quick experiment you can do is to once do the preprocessing steps that you usually do and then feed it to the model and get the results. And once feed the dataset as it is to the model to compare the difference. In my experience doing the preprocessing won't make any difference, based on the dataset it gave me 1 more or less percent difference in accuracy (...


2

FunctionTransformer is useful because it allows you to apply a custom function in a pipeline. Because Pipeline() from sklearn.pipeline only works with objects that implement the .transform() and .fit() methods, you use FunctionTransformer to change your custom function to allow .transform() and/or .fit() to be used on it. You could transform a DataFrame ...


2

I usually replace unseen and NaN values with the global target mean. There are also already implemented transformers for target encoding that you could use that gives you some options such as smoothing: scikit contrib - target encoder.


2

class StrayCommaRemover(TransformerMixin): def __init__(self): //Initialize self by setting some variables here which can be passed as a input to transformer def fit(self, X, y=None): self.columns = X.columns //Setting context based on input Data return self def transform(self, X, y=None): // Actual transformation logic X= X.withColumn('...


2

Remember that BERT was first pre-trained using the concatenation of BooksCorpus (800M words) and English Wikipedia (2,500M words). Then the fine-tuning in your case uses the SQuAD dataset consisting of 100,000+ questions (based on Wikipedia articles) with a learning rate in the order of e-5. So when you add let's say 100 new domain-specific question/...


2

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input. In order not to use a sequence of integers (1, 2, 3, ... n) because of the lack of boundary in the value and the magnitude, a float friendly is preferred. But, just using a limited (0 to 1) option means you ...


2

Its all ok. Exactly because your values are between 1 and 2 does he chooses 1.5. In other words trees will select 1 and 2 as values (integers not real numbers) but 1.5 is there as cut-off point between these classes.


2

To include this logic into a pipeline you have to create a custom transformer. You need to ask yourself: [INIT] Are there any parameters in my logic? The variable you want to impute and the category you want this imputation to be based on. [FIT] What part of the logic is related to computing what the transformation will be? When you compute the median()...


2

So, the question talks about how to treat transformation choices as hyper parameters. How I would go about it is the following: Use one baseline model architecture for the data and then repeat the following: Instantiate the baseline model (effectively make sure all of the weights are initialised) Create the transformed dataset Train the model Compute ...


1

Firstly, fixing the input size of a model is more of an architectural decision than a problem. By fixing input size we: Limit the GPU memory usage while training Reduce the training time per epoch Reduce evaluation time per input sample If you want to, you can train your own transformer model by increasing the size of the input or increasing the number of ...


1

To answer your specific questions: AdditiveAttention() and Attention() layers, are (loosely but not exactly) based on Bahdanau and Luong's attentions, respectively. They use post-2018 semantics of queries, values and keys. To map the semantics to the Bahdanau or Luong's paper, you can consider the 'query' to be the last decoder hidden state. The 'values' ...


1

The general formulation of attention with queries, keys and values corresponds to a re-retrieval view on attention: you have some queries that you use to retrieve some values based on keys that correspond to them. With RNNs, attention is used for sequence-to-sequence models like machine translation. (Time series forecasting is usually formulated as sequence ...


1

The ranking of the answers is part of the ML process, i.e. a system should be trained to rank the answers according to their relevance. Heuristic measures such as the ones mentioned in your question may offer decent approximations, but as you noticed they are very limited. You may be interested in datasets and methods used in shared tasks about QA, for ...


Only top voted, non community-wiki answers of a minimum length are eligible