I'm trying to train an autoencoder (in PyTorch) to reconstruct gene profiles. At the moment I'm using the Mean Squared Error (MSE) loss for training: the model is not overfitting and both the training and validation loss are decreasing. The problem is that the cosine similarity on the validation set between original and reconstructed vectors has a mean of 0.4. I was thinking of using the cosine similarity as loss function instead of MSE.
At the following link (slide 18), the author proposes the following loss:
$$ l(x_1, x_2, y) = \begin{cases} max(0, cos(x_1, x_2) - m) & \text{if $y$ == -1} \\ 1 - cos(x_1, x_2) & \text{if $y$ == 1}. \end{cases} $$
I'm not entirely sure whether this is the right approach, but I'm having some difficulties even understanding the formula. What is $y$ (the cosine similarity between $x_1$ and $x_2$?) and why it's an input to the loss?