1
$\begingroup$

In the paper GloVe: Global Vectors for Word Representation, there is this part (bottom of third page) I don't understand:

enter image description here

I understand what groups and homomorphisms are. What I don't understand is what requiring $ F $ to be a homomorphism between $ (\mathbb{R},+) $ and $ (\mathbb{R}_{>0},\times) $ has to do with making $ F $ symmetrical in $ w $ and $ \tilde{w}_k $.

Am I misunderstanding something? We want $ F $ to be unchanged if we either interchange $ w_i $ and $ \tilde{w}_k $ OR interchange $ w_j $ and $ \tilde{w}_k $, right? Is this the only way to achieve the symmetry between $ w $ and $ \tilde{w}_k $?

$\endgroup$
1
$\begingroup$

If you're asking if the group homomorphism makes the the process symmetric then no it doesn't directly. However, they use the fact that they require a group homomorphism to show that $w_{i}^{T} \tilde{w}_k = log(P_{ik})=log(X_{ik}) - log(X_{i})$ This nearly gives us symmetry. Finally by adding $\tilde{b}_{k}$ into the equation you restore symmetry.

So in short $w_{i}^{T} \tilde{w}_k + b_{i} + \tilde{b}_{k} = log(X_{ik})$ is what ensures symmetry, and the group homomorphism is a tool to get there.

Update:

Some more details Essentially, what we want is the ability to peform a label switch. Group homomorphism helps with this process because it perseves a mapping between the $(R, +)$ and $(R, x)$.

$F((w_{i}^{T} - w_{j}^{T})w_{k}^{'})=F(w_{i}^{T}w_{k}^{'}+( - w_{j}^{T}w_{k}^{'})) = F(w_{i}^{T}w_{k}^{'}) \times F(-w_{j}^{T}w_{k}^{'} )= F(w_{i}^{T}) \times F(w_{j}^{T}w_{k}^{'})^{-1} = \frac{F(w_{i}^{T}w_{k}^{'})}{F(w_{j}^{T}w_{k}^{'})}$

The group homomorphism here allows for that to occur. Therefore we can see that by setting $F(w_{i}^{T}w_{k}^{i}) = \frac{X_{ik}}{X_{i}}$

Now finally we can say that $w_{i}^{T} {w}_k^{'} = log(P_{ik})=log(X_{ik}) - log(X_{i}).$

So as far as your comment, it is the most sensible chocie for their method and of which they buld the core mathematicals to GloVE. Changing it, I imagine wouldn't be a trivial thing. I imagine if you did, much of what is derived, including the loss function would change. But with that said, I imagine there are otherwise to achieve label switching.

$\endgroup$
  • $\begingroup$ So this is a 'sensible choice' on their part, right? It's not strictly necessary for F to be that homomorphism for there to be symmetry in w and w~, right? $\endgroup$ – Noppawee Apichonpongpan Jan 25 '18 at 16:31
  • $\begingroup$ Depends what you mean by sensible. It's sensible because it gives them their property, but with that said, I would also state that it's a core property of their method. $\endgroup$ – Tophat Jan 25 '18 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.