1
$\begingroup$

I am trying to write a model that has the input vector of one embedding (say $E_1$) and predicts the corresponding vector in the second embedding $E_2$. Both are n-dimensional real dense vectors $\mathbb{R}^n$.

Concretely one is a skipgram word embedding, and the other is a node2vec graph embedding. I have approximately 30 000 training examples that provide a mapping between the two. Since they are both just real vectors, it seems like a trivial task to write a simple MLP that learns the non-linear transformation of one to the other (I actually don't really care about over fitting here, since the domain is closed).

However, I can't seem to get it to work properly.

In Keras, naively something like this should work:

in = Input(e1_dim)
hidden = Dense(some_value, activation="tanh")(in)
out = Dense(e2_dim)(hidden)

I have tried adding more hidden layers, but I think my problem lies with the fact that the input and output vectors are in the domain (-1,1), and so the choice of initializer, loss function, and activation function is critical.

I have tried setting the initializer to RandomUniform, but still no good results. For loss I have tried MAE en cosine_proximity, but both seem to produce terrible results. In particular cosine_proximity seems to not get above -0.5 which might be a sign. Any thoughts on the choice of architecture and loss function for mapping one embedding onto another (essentially a high-dimensional non-linear regression?)

$\endgroup$
1
$\begingroup$

You should take into account a few factors:

  • some_value is key here. You should probably choose a value greater than e1_dim and e2_dim. This may be a problem due to the large memory needs for high embedding dimensionalities.
  • if e2_dim is bounded to $(-1,+1)$, you may want to set a tanh activation in the last layer (i.e. after Dense(e2_dim)). With that in place, you may want to replace the activation of the hidden layer with a ReLU, to avoid squashing the gradients unnecessarily.
  • If your embeddings are directional, or they are in the unit ball, then you should be fine with cosine_proximity. Maybe a better option would be to use the von Mises-Fischer distance, but I think it's not straightforward. Otherwise, you may try World-Mover's Distance.

That being said, a lot of embedded space mapping works out there assume that the embedded spaces are approximately isomorphic and just go ahead with a linear transformation. You may want to have a look at the relevant bilingual embedding mapping literature; you may start with the work by Artetxe et al., 2016. I don't know if the mentioned assumption is met to any degree in your scenario (skipgram $\leftrightarrow$ node2vec), but you may assess this with the VF2 algorithm, like Artetxe et al. did.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.