Mapping one embedding to another using Deep Learning

Question

I am trying to write a model that has the input vector of one embedding (say $E_1$) and predicts the corresponding vector in the second embedding $E_2$. Both are n-dimensional real dense vectors $\mathbb{R}^n$.

Concretely one is a skipgram word embedding, and the other is a node2vec graph embedding. I have approximately 30 000 training examples that provide a mapping between the two. Since they are both just real vectors, it seems like a trivial task to write a simple MLP that learns the non-linear transformation of one to the other (I actually don't really care about over fitting here, since the domain is closed).

However, I can't seem to get it to work properly.

In Keras, naively something like this should work:

in = Input(e1_dim)
hidden = Dense(some_value, activation="tanh")(in)
out = Dense(e2_dim)(hidden)

I have tried adding more hidden layers, but I think my problem lies with the fact that the input and output vectors are in the domain (-1,1), and so the choice of initializer, loss function, and activation function is critical.

I have tried setting the initializer to RandomUniform, but still no good results. For loss I have tried MAE en cosine_proximity, but both seem to produce terrible results. In particular cosine_proximity seems to not get above -0.5 which might be a sign. Any thoughts on the choice of architecture and loss function for mapping one embedding onto another (essentially a high-dimensional non-linear regression?)

Answer 1

You should take into account a few factors:

• some_value is key here. You should probably choose a value greater than e1_dim and e2_dim. This may be a problem due to the large memory needs for high embedding dimensionalities.
• if e2_dim is bounded to $(-1,+1)$, you may want to set a tanh activation in the last layer (i.e. after Dense(e2_dim)). With that in place, you may want to replace the activation of the hidden layer with a ReLU, to avoid squashing the gradients unnecessarily.
• If your embeddings are directional, or they are in the unit ball, then you should be fine with cosine_proximity. Maybe a better option would be to use the von Mises-Fischer distance, but I think it's not straightforward. Otherwise, you may try World-Mover's Distance.

That being said, a lot of embedded space mapping works out there assume that the embedded spaces are approximately isomorphic and just go ahead with a linear transformation. You may want to have a look at the relevant bilingual embedding mapping literature; you may start with the work by Artetxe et al., 2016. I don't know if the mentioned assumption is met to any degree in your scenario (skipgram $\leftrightarrow$ node2vec), but you may assess this with the VF2 algorithm, like Artetxe et al. did.