In BERT, They replace separator and start of sentence with special token labels. What are there corresponding values in embedding_matrix. Are they 0-vector?

I wanted to replace the proper nouns like names, buildings, locations with similar approach. How should i go about masking the same?


First a clarification: there is no masking at all in the [CLS] and [SEP] tokens. These are artificial tokens that are respectively inserted before the first sequence of tokens and between the first and second sequences.

About the value of the embedded vectors of [CLS] and [SEP]: they are not filled with 0's but contain numerical representations like any of the other vectors in the embedding table. From the point of view of the model, there is nothing special about their embedded vectors, so they are trained normally like the others.

About masking proper names: you should take into account that BERT is a subword-based model. This means that words get split into subwords that are part of the vocabulary. Sometimes a word maps to only one token, but other times a single word maps to a sequence of several tokens. Infrequent words tend to be sliced almost at the character level. Depending on the data the model is trained with, the proper nouns you are referring to may be split into multiple subwords. You should take this into account when designing your masking procedure by either:

  • Before tokenizing, you replace the proper nouns with a text sequence that is included in the vocabulary and that is exclusive to them.
  • After tokenizing the original sentence, you check if the text contained a proper noun and if so, you locate the sequence of subwords associated to them and replace each subsequence with a special proper noun token ID.

Take into account that this is not similar to BERT's inherent masking, where the input token is masked but the loss forces the model to predict the original token. In your case, the original proper noun would be masked to never return.

  • $\begingroup$ How does the [CLS] embedding get learned during the self-supervised training if it is never masked like the other tokens? Is it only useful for classification tasks after pretraining? $\endgroup$
    – Evan Zamir
    Aug 19 at 5:32
  • $\begingroup$ Hi @EvanZamir, can you please post a new question with these doubts? $\endgroup$
    – noe
    Aug 19 at 11:41

I think replacing Proper Nouns with an aggregated Proper Nouns vector should do the trick

Basically, words like Barcelona, Spain, India, and other locations will have a high similarity to a bias vector, we can find out that vector using standard deviation of this matrix along the column axis. Whichever has a low value can be retained, rest all can be set to 0

eg: Delhi, can be replace with [2,3,4,0,0,0...] where [2,3,4...] are common attributes of other locations


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.