It is likely to be independent of dictionary. Loading BERT model and running a forward pass has its own memory requirements. How did you figure that it is memory leak?
Try visualizing memory footprints on each step in your code by using some break points. It will give you clear idea about the hardware requirements and memory leak if any.
With regard to a dictionary of words, there can be no single dictionary for BERT because the BERT embeddings incorporate contextual information (i.e. the surrounding words in the sentence change the embedding for your target word). In theory, you could construct a dictionary for your words by passing single word sentences (though a single word may be broken ...
Translation as a pre-processing step is usually sufficient for many tasks (e.g. sentiment classification), but naturally undesirable for other tasks e.g. grading someone in written Dutch fluency.
Hence, for these tasks, the objective is:
Be able to train a language model for your specific language
However, you want to be able to do this with minimal ...
BERT does not give word representations, but subword representations (see this). Nevertheless, it is common to average the representations of the subwords in a word to obtain a "word-level" representation.
You may try to handle this as a normal tagging problem, where the tag of each word is the class associated with the word, much like part-of-...
In this context, masking means replacing the token with a special [MASK] token. The network does not have the information of what the original token was, the only way how it could potentially figure out what it was is by looking at the context.
It is not the loss function that guarantees that the model learns something meaningful, it is architecture design ...
For your first question, you can check if the tokenizer covers a certain string with the following:
text = 'today is a good day 😃'
ids2string = lambda ids: tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(ids))
> <s>today is a good day 😃</s>
If emoji is not included in the tokenizer ...
The transformers library uses complex output objects instead of plain tuples as return type since one of the updates after 3.5.1.:
from transformers import BertModel, BertTokenizer
t = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
o = t.encode_plus('this is a sample sentence', return_tensors='pt')
Disclaimer: this answer might be disappointing ;)
In general my advice would be to carefully analyze the errors that the model makes and try to make the model deal with these cases better. This can involve many different strategies depending on the task and the data. Here are a few general directions to consider:
Most of the time the imbalance is not the ...
I don't know about any specific recommendation related to BERT, but my general advice is this:
Do not to systematically use oversampling when the data is imbalanced, at least not before specifically identifying performance issues caused by the imbalance. I see many questions here on DataScienceSE about solving problems which are caused by using oversampling ...
It is the encoder part of the Transformer model that is bidirectional in nature, not the whole model.
The full Transformer model has two parts: encoder and decoder. This encoder-decoder model is used for sequence-to-sequence tasks, like machine translation.
There are other tasks, however, that do not need the full model, but only one of its parts. For ...