1
$\begingroup$

Presume I have a vector space, and I am attempting to compress it into a latent vector space, while minimizing error in cosine similarity between entries. Suppose that I know the actual cosine similarity scores between the latent space. i.e.

Let $X \subset \mathbb{R}^n$ be my set of vectors, and $r: X \times X \to [-1, 1]$ be a function. I am trying to find a function $f: X \to \mathbb{R}^m$ such that the error $$\sum_{(x, y) \in X \times X} |sim(f(x), f(y)) - r(x, y)|$$ is minimized.

What is a good way of finding $f$ given $X$, $r$ and $m$? I am comfortable using gradient descent or neural networks.

$\endgroup$

1 Answer 1

1
$\begingroup$

To find the necessary function using error back propagation, you only need to design the architecture of the neural network, which will represent the desired function (for example, several layers, activation functions, etc.).

I do not see any problems when using the function you specified as a loss function. Because cosine similarity in this case is the product vector embeddings, then the gradient is calculated obviously.

I recommend that you study how face recognition models are trained (for example, Facenet). Very similar approaches are used there.

I'm pretty sure you're aware of the use of Kullback–Leibler divergence to solve this problem. I just wanted to draw attention to the possible use of SGD for its optimization, which may give a hint in your case.

$\endgroup$
1
  • 1
    $\begingroup$ I think I had a lapse in reason! You mentioned that the gradient is obvious because of the product of vector embedding, and yes. You're right. Seeing two variables within a loss function worried me, but in this case it is trivial. Thank you very much. $\endgroup$ Dec 7, 2023 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.