I would like to create a loss function that encourages the output of the embedding of an autoencoder to be dense. I don't have an explicit condition for how density is defined, but one option would be that the average distance between a point and its k nearest neighbors should be minimized, where k is some parameter. This is balanced against the reconstruction error, so the embedded points won't all converge to the same value. Is there a way to construct a loss function of this, or something similar, in a way that allows the average density around each point to be backprogated through the autoencoder?
Very good question!
An approach towards your goal is Center loss. The loss rewards if points of a class gather around a center. You can adapt this approach for your Autoencoder.
You can use the nearest neighbor loss directly. You can backpropagate through that, though its gradient isn't very stable.
Perhaps better is to use a soft nearest neighbor loss, e.g., see Salakhutdinov & Hinton, "Learning a nonlinear embedding by preserving class neighbourhood structure", 2007. This gives a more stable gradient.
Both of these will be very slow, because they require comparing each point to every other point in the training set. A hack to speed it up is to only compare to other samples in the same mini-batch.
There may be many other approaches as well.