I would like to create a loss function that encourages the output of the embedding of an autoencoder to be dense. I don't have an explicit condition for how density is defined, but one option would be that the average distance between a point and its k nearest neighbors should be minimized, where k is some parameter. This is balanced against the reconstruction error, so the embedded points won't all converge to the same value. Is there a way to construct a loss function of this, or something similar, in a way that allows the average density around each point to be backprogated through the autoencoder?
1
-
$\begingroup$ What do you mean by dense? How would you tell whether a proposed solution is effective or useful or meets your needs? It seems that before you can explore solutions, you first need to figure out how to formulate the problem. $\endgroup$– D.W.Commented Dec 13, 2019 at 2:11
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Very good question!
An approach towards your goal is Center loss. The loss rewards if points of a class gather around a center. You can adapt this approach for your Autoencoder.
$\begingroup$
$\endgroup$
You can use the nearest neighbor loss directly. You can backpropagate through that, though its gradient isn't very stable.
Perhaps better is to use a soft nearest neighbor loss, e.g., see Salakhutdinov & Hinton, "Learning a nonlinear embedding by preserving class neighbourhood structure", 2007. This gives a more stable gradient.
Both of these will be very slow, because they require comparing each point to every other point in the training set. A hack to speed it up is to only compare to other samples in the same mini-batch.
There may be many other approaches as well.
