BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?

Question

Word2vec: Word2vec provides a vector for each token/word and those vectors encode the meaning of the word. Although those vectors are not human interpretable, the meaning of the vectors are understandable/interpretable by comparing with other vectors (for example, the vector of dog will be most similar to the vector of cat), and various interesting equations (for example king-men+women=queen, which proves how well those vectors hold the semantic of words).

The problem with word2vec is that each word has only one vector but in the real world each word has different meaning depending on the context and sometimes the meaning can be totally different (for example, bank as a financial institute vs bank of the river).

Bert: One important difference between Bert/ELMO (dynamic word embedding) and Word2vec is that these models consider the context and for each token, there is a vector.

Now the question is, do vectors from Bert hold the behaviors of word2Vec and solve the meaning disambiguation problem (as this is a contextual word embedding)?

Experiments To get the vectors from google's pre-trained model, I used bert-embedding-1.0.1 library. I first tried to see whether it hold the similarity property. To test, I took the first paragraphs from wikipedia page of Dog, Cat, and Bank (financial institute). The similar word for dog is: ('dog',1.0) ('wolf', 0.7254540324211121) ('domestic', 0.6261438727378845) ('cat', 0.6036421656608582) ('canis', 0.5722522139549255) ('mammal', 0.5652133226394653) Here, the first element is token and second is the similarity.

Now for disambiguation test: Along with Dog, Cat and Bank (financtial institute), I added a paragraph of River bank from wikipedia. This is to check that bert can differentiate between two different types of Bank. Here the hope is, the vector of token bank (of river) will be close to vector of river or water but far away from bank(financial institute), credit, financial etc. Here is the result: The second element is the sentence to show the context.

('bank', 'in geography , the word bank generally refers to the land alongside a body of water . different structures are referred to as', 1.0)
('bank', 'a bank is a financial institution that accepts deposits from the public and creates credit .', 0.7796692848205566)
('bank', 'in limnology , a stream bank or river bank is the terrain alongside the bed of a river , creek , or', 0.7275459170341492)
('bank', 'in limnology , a stream bank or river bank is the terrain alongside the bed of a river , creek , or', 0.7121304273605347)
('bank', 'the bank consists of the sides of the channel , between which the flow is confined .', 0.6965076327323914)
('banks', 'markets to their importance in the financial stability of a country , banks are highly regulated in most countries .', 0.6590269804000854)
('banking', 'most nations have institutionalized a system known as fractional reserve banking under which banks hold liquid assets equal to only a', 0.6490173935890198)
('banks', 'most nations have institutionalized a system known as fractional reserve banking under which banks hold liquid assets equal to only a', 0.6224181652069092)
('financial', 'a bank is a financial institution that accepts deposits from the public and creates credit .', 0.614281952381134)
('banks', 'stream banks are of particular interest in fluvial geography , which studies the processes associated with rivers and streams and the deposits', 0.6096583604812622)
('structures', 'in geography , the word bank generally refers to the land alongside a body of water . different structures are referred to as', 0.5771245360374451)
('financial', 'markets to their importance in the financial stability of a country , banks are highly regulated in most countries .', 0.5701562166213989)
('reserve', 'most nations have institutionalized a system known as fractional reserve banking under which banks hold liquid assets equal to only a', 0.5462549328804016)
('institution', 'a bank is a financial institution that accepts deposits from the public and creates credit .', 0.537483811378479)
('land', 'in geography , the word bank generally refers to the land alongside a body of water . different structures are referred to as', 0.5331911444664001)
('of', 'in geography , the word bank generally refers to the land alongside a body of water . different structures are referred to as', 0.527492105960846)
('water', 'in geography , the word bank generally refers to the land alongside a body of water . different structures are referred to as', 0.5234918594360352)
('banks', 'bankfull discharge is a discharge great enough to fill the channel and overtop the banks .', 0.5213838815689087)
('lending', 'lending activities can be performed either directly or indirectly through due capital .', 0.5207482576370239)
('deposits', 'a bank is a financial institution that accepts deposits from the public and creates credit .', 0.5131596922874451)
('stream', 'in limnology , a stream bank or river bank is the terrain alongside the bed of a river , creek , or', 0.5108630061149597)
('bankfull', 'bankfull discharge is a discharge great enough to fill the channel and overtop the banks .', 0.5102289915084839)
('river', 'in limnology , a stream bank or river bank is the terrain alongside the bed of a river , creek , or', 0.5099104046821594)

Here, the result of the most similar vectors of bank (as a river bank, the token is taken from the context of the first row and that is why the similarity score is 1.0. So, the second one is the closest vector). From the result, it can be seen that the first most close token's meaning and context is very different. Even the token river, water andstream` has lower similarity.

So, it seems that the vectors do not really disambiguate the meaning. Why is that? Isn't the contextual token vector supposed to disambiguate the meaning of a word?

Answer 1

BERT and ELMo are recent advances in the field. However, there is a fine but major distinction between them and the typical task of word-sense disambiguation: word2vec (and similar algorithms including GloVe and FastText) are distinguished by providing knowledge about the constituents of the language. They provide semantic knowledge, typical about word types (i.e. words in the dictionary), so they can tell you something about the meaning of words like "banana" and "apple".

However, this knowledge is at the level of the prototypes, rather than their individual instances in texts (e.g. "apple and banana republic are american brands" vs "apple and banana are popular fruits"). The same embedding will be used for all instances (and all different senses) of the same word type (string).

In past years, distributional semantics methods were used to enhance word embeddings to learn several different vectors for each sense of the word, such as Adaptive Skipgram. These methods follow the approach of word embeddings, enumerating the constituents of the language, but just at a higher resolution. You end up with a vocabulary of word-senses, and the same embedding will be applied to all instances of this word-sense.

BERT and ELMo represent a different approach. Instead of providing knowledge about the word types, they build a context-dependent, and therefore instance-specific embedding, so the word "apple" will have different embedding in the sentence "apple received negative investment recommendation" vs. "apple reported new record sale". Essentially, there is no embedding for the word "apple", and you cannot query for "the closest words to apple", because there are infinite number of embeddings for each word type.

Thus, BERT and ELMo handle the differences between word-senses but will not reveal what are the different senses of the word.

Answer 2

I think there are a few misconceptions in your statements. Please take into account the following

• BERT does not provide word-level representation. It provides sub-words embeddings and sentence representations. For some words, there may be a single subword while, for others, the word may be decomposed in multiple subwords. The representations of subwords cannot be combined into word representations in any meaningful way.

• ELMO does provide word-level representations.

• Word embeddings don't reflect meaning but co-occurrence statistics within the context window. This means that two antonyms will probably have similar representations, as they can appear in similar contexts within short context windows.

• Distances over continuous representation spaces do not mix well with untangled concepts, like meaning, e.g. should two antonyms be closer than two unrelated words?

That being said...

• If you want a model that does word sense disambiguation (WSD), you should train a model on a WSD dataset if there is available data.
• If you have no data, you could try a nearest neighbor approach like in this article, where the authors specifically show their approach with "bank" for different contextualized word embeddings (see figure below). You should also have a look at this other article, where they also study the geometry of BERT representations in relation to WSD.

Answer 3

The most important question to ask is : for which purpose do you need that?

1. You are not right when claiming that Word2Vec creates vectors for words without taking into account the context. The vectors are created in fact using a window (the size of the windo is one of the settings in W2V) in order to get the neighbouring words, so yes, it takes into account the context! That is why you can have those famous equations : king-man = queen-woman, high cosine similarity between two synonyms or words having the same part of speech, etc, etc.

2. If you wish to have two meanings, why not create two models, with data from two separate sectors (banking and general, let us say). In this case, each model will recognize bank with a different meaning.

Once again, what exactly is you need here ?

Answer 4

One possible way to disambiguate multiple meanings for a word is to modify the string literal during training.

For bank, the model would learn bank_FINANCE and separately learn bank_RIVER. Creating separate tokens from the string literal is a way to allow the tokens to capture different meanings and disambiguate those separate meanings.

This is commonly done for parts-of-speech, excuse can be excuse_NOUN and excuse_VERB.

Adding context-specific metadata to disambiguate the meaning of the same string literal would work in a similar way for both word2vec and BERT.

Answer 5

One option is to add an additional neural network model from the output of standard BERT. During training, standard BERT would learn the sentence embeddings. The additional neural network would learn to classify the different senses of the same string literal based on the context of the sentence.

One version of this was done in "Using BERT for Word Sense Disambiguation".