Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
In BERT, They replace separator and start of sentence with special token labels. What are there corresponding values in embedding_matrix. Are they 0-vector?
I wanted to replace the proper nouns like names, buildings, locations with similar approach. How should i go about masking the same?
First a clarification: there is no masking at all in the [CLS] and [SEP] tokens. These are artificial tokens that are respectively inserted before the first sequence of tokens and between the first and second sequences.
About the value of the embedded vectors of [CLS] and [SEP]: they are not filled with 0's but contain numerical representations like any of the other vectors in the embedding table. From the point of view of the model, there is nothing special about their embedded vectors, so they are trained normally like the others.
About masking proper names: you should take into account that BERT is a subword-based model. This means that words get split into subwords that are part of the vocabulary. Sometimes a word maps to only one token, but other times a single word maps to a sequence of several tokens. Infrequent words tend to be sliced almost at the character level. Depending on the data the model is trained with, the proper nouns you are referring to may be split into multiple subwords. You should take this into account when designing your masking procedure by either:
Take into account that this is not similar to BERT's inherent masking, where the input token is masked but the loss forces the model to predict the original token. In your case, the original proper noun would be masked to never return.
I think replacing Proper Nouns with an aggregated Proper Nouns vector should do the trick
Basically, words like Barcelona, Spain, India, and other locations will have a high similarity to a bias vector, we can find out that vector using standard deviation of this matrix along the column axis. Whichever has a low value can be retained, rest all can be set to 0
eg: Delhi, can be replace with [2,3,4,0,0,0...] where [2,3,4...] are common attributes of other locations
