1
$\begingroup$

Word2vec: Word2vec provides a vector for each token/word and those vectors encode the meaning of the word. Although those vectors are not human interpretable, the meaning of the vectors are understandable/interpretable by comparing with other vectors (for example, the vector of dog will be most similar to the vector of cat), and various interesting equations (for example king-men+women=queen, which proves how well those vectors hold the semantic of words).

The problem with word2vec is that each word has only one vector but in the real world each word has different meaning depending on the context and sometimes the meaning can e totally different (for example, bank as a financial institute vs bank of the river).

Bert: One important difference between Bert/ELMO (dynamic word embedding) and Word2vec is that these models consider the context and for each token, there is a vector.

Now the question is, do vectors from Bert hold the behaviors of word2Vec and solve the meaning disambiguation problem (as this is a contextual word embedding)?

Experiments To get the vectors from google's pre-trained model, I used bert-embedding-1.0.1 library. I first tried to see whether it hold the similarity property. To test, I took the first paragraphs from wikipedia page of Dog, Cat, and Bank (financial institute). The similar word for dog is: ('dog',1.0) ('wolf', 0.7254540324211121) ('domestic', 0.6261438727378845) ('cat', 0.6036421656608582) ('canis', 0.5722522139549255) ('mammal', 0.5652133226394653) Here, the first element is token and second is the similarity.

Now for disambiguation test: Along with Dog, Cat and Bank (financtial institute), I added a paragraph of River bank from wikipedia. This is to check that bert can differentiate between two different type of Bank. Here the hope is, the vector of token bank (of river) will be close to vector of river or water but far away from bank(financial institute), credit, financial etc. Here is the result: The second element is the sentence to show the context.

('bank', 'in geography , the word bank generally refers to the land alongside a body of water . different structures are referred to as', 1.0)
('bank', 'a bank is a financial institution that accepts deposits from the public and creates credit .', 0.7796692848205566)
('bank', 'in limnology , a stream bank or river bank is the terrain alongside the bed of a river , creek , or', 0.7275459170341492)
('bank', 'in limnology , a stream bank or river bank is the terrain alongside the bed of a river , creek , or', 0.7121304273605347)
('bank', 'the bank consists of the sides of the channel , between which the flow is confined .', 0.6965076327323914)
('banks', 'markets to their importance in the financial stability of a country , banks are highly regulated in most countries .', 0.6590269804000854)
('banking', 'most nations have institutionalized a system known as fractional reserve banking under which banks hold liquid assets equal to only a', 0.6490173935890198)
('banks', 'most nations have institutionalized a system known as fractional reserve banking under which banks hold liquid assets equal to only a', 0.6224181652069092)
('financial', 'a bank is a financial institution that accepts deposits from the public and creates credit .', 0.614281952381134)
('banks', 'stream banks are of particular interest in fluvial geography , which studies the processes associated with rivers and streams and the deposits', 0.6096583604812622)
('structures', 'in geography , the word bank generally refers to the land alongside a body of water . different structures are referred to as', 0.5771245360374451)
('financial', 'markets to their importance in the financial stability of a country , banks are highly regulated in most countries .', 0.5701562166213989)
('reserve', 'most nations have institutionalized a system known as fractional reserve banking under which banks hold liquid assets equal to only a', 0.5462549328804016)
('institution', 'a bank is a financial institution that accepts deposits from the public and creates credit .', 0.537483811378479)
('land', 'in geography , the word bank generally refers to the land alongside a body of water . different structures are referred to as', 0.5331911444664001)
('of', 'in geography , the word bank generally refers to the land alongside a body of water . different structures are referred to as', 0.527492105960846)
('water', 'in geography , the word bank generally refers to the land alongside a body of water . different structures are referred to as', 0.5234918594360352)
('banks', 'bankfull discharge is a discharge great enough to fill the channel and overtop the banks .', 0.5213838815689087)
('lending', 'lending activities can be performed either directly or indirectly through due capital .', 0.5207482576370239)
('deposits', 'a bank is a financial institution that accepts deposits from the public and creates credit .', 0.5131596922874451)
('stream', 'in limnology , a stream bank or river bank is the terrain alongside the bed of a river , creek , or', 0.5108630061149597)
('bankfull', 'bankfull discharge is a discharge great enough to fill the channel and overtop the banks .', 0.5102289915084839)
('river', 'in limnology , a stream bank or river bank is the terrain alongside the bed of a river , creek , or', 0.5099104046821594)

Here, the result of the most similar vectors of bank (as a river bank, the token is taken from the context of the first row and that is why the similarity score is 1.0. So, the second one is the most close vector). From the result it can be seen that the first most close token's meaning and context is very different. Even the token river, water andstream` has lower similarity.

So, it seems that, the vectors does not really disamguate the meaning. Why is that? is not the contextual token vector supposed to disambiguate the meaning of word?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.