I'm a layman making a foray into NLP and I have a question: The landmark paper A Neural Probabilistic Language Model (Bengio et al., 2003) makes an attempt at statistical language modelling by (1) learning a distributed word feature vector for every word (i.e. a word embedding in contemporary terminology) and (2) feeding those word vectors into a neural net to predict the successor to an n-gram of words.
The learned word vectors preserve similarity in the sense, that word vectors of words which occur in the same context during training tend to be closer together. This allows to "fight the curse of dimensionality with its own weapons", as the authors put it poetically, since "Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence".
Here's what's causing me headache: I fail to understand how the training process of this model produces word vectors which do preserve similarity in the sense above, instead of learning, well, not so good word vectors. I don't see where this extra constraint of "learn word vectors, but preserve similarity" is respected during training. The learning process described looks like a regular backprop without any extra effort put into learning good word vectors.
I've been staring at the paper for hours and I just don't get it. No publications or websites I've come across that discuss this paper mention my conundrum, so I assume it must be something simple that I'm overlooking. May somebody kindly help me out?