1
$\begingroup$

I'm a layman making a foray into NLP and I have a question: The landmark paper A Neural Probabilistic Language Model (Bengio et al., 2003) makes an attempt at statistical language modelling by (1) learning a distributed word feature vector for every word (i.e. a word embedding in contemporary terminology) and (2) feeding those word vectors into a neural net to predict the successor to an n-gram of words.

The learned word vectors preserve similarity in the sense, that word vectors of words which occur in the same context during training tend to be closer together. This allows to "fight the curse of dimensionality with its own weapons", as the authors put it poetically, since "Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence".

Here's what's causing me headache: I fail to understand how the training process of this model produces word vectors which do preserve similarity in the sense above, instead of learning, well, not so good word vectors. I don't see where this extra constraint of "learn word vectors, but preserve similarity" is respected during training. The learning process described looks like a regular backprop without any extra effort put into learning good word vectors.

I've been staring at the paper for hours and I just don't get it. No publications or websites I've come across that discuss this paper mention my conundrum, so I assume it must be something simple that I'm overlooking. May somebody kindly help me out?

Improve this question
$\endgroup$
2
  • $\begingroup$ Welcome to DataScienceSE. I'm not sure that I understand your question. I assume that you're familiar with the distributional semantics assumption that meaning is represented by the context words, right? Normally a reasonably good semantic representation is learned by generalizing across multiple occurrences of a target word, thus forming a context distribution. Of course it doesn't work that well for rare words. But maybe this is is not at all your question ;) $\endgroup$
    – Erwan
    Aug 1, 2022 at 18:09
  • $\begingroup$ @Erwan I'm indeed familiar with the distributional hypothesis :) My question is much more superficial: We apply gradient descent to optimize a function composed of two parts, A and B. Why does A turn out to have special qualities without us having put any effort into achieving this? $\endgroup$
    – toughkip
    Aug 2, 2022 at 7:57

2 Answers 2

Reset to default
1
$\begingroup$

It's because during the training, it is not only the neural network with tanh that learns but also the words' representation in Matrix C.

There are 2 different parts in this model:

  • The classic neural network can make correlations between words sequentially.
  • The words' representation can make correlations between words in general. Without words' representation, the model couldn't make unknown correlations that haven't been learned previously. They've explained it in the paper with the words "cat" and "dog". This is obtained progressively by building a map of probability between words.

For instance: The sentences

"The cat is walking in the room." and

"The dog is running in the bedroom."

would increase the neighbor vectors (ex: "walking", "running", "room", "bedroom") in Matrix C of both "cat" and "dog", thanks to the phrases' similarity.

In other words, if we increase the vector of "cat"/"walking", it would also increase the vectors of "cat"/"running" and "dog"/"walking" in a similar context, thanks to an embedding lookup.

We could define the words' representation as a general probability space of all words, which is slightly modified at each learning iteration.

Explanation from Yoshua B (great thanks):

The reason why similar words end up having similar word embeddings is because of the smoothness of the neural net that takes these word embeddings in input. If "The cat is walking in the --- " can be completed by "room", it is also true when we replace "cat" by "dog", which puts pressure on both words to have similar word embeddings. Small change of the embeddings = small change in the output probabilities.

If you think about it, the architecture is very similar to a 1-D convolutional neural network:

  • the matrix C corresponds to a usual dot-product neural operation when the input is a one-hot vector for the word at each position (with a 1 at the position corresponding to the word symbol)
  • using the same matrix C at every position makes sense and proceeds of the same inductive bias as in 1-D convolutional neural networks: the meaning of a word is position invariant (i.e. if we only know that a word appeared at position 3 vs 4, the meaning does not change).

The idea of using such layers, with shared weights across different positions, is found not just in 1-D convnets and time-delay neural net (which pre-existed the NNLM, and which I worked on in my 1991 PhD thesis), but also in neural nets operating on symbols (which were explored among others by Geoff Hinton and his student Paccanaro a few years earlier, cited in the paper).

Improve this answer
$\endgroup$
8
  • $\begingroup$ That's all clear. What I don't understand is why the learning process produces good word vectors. Gradient descent trains the whole model and it just so happens that the first part of the model produces word vectors with a wonderfully benevolent quality (similarity as mentioned above). Why? $\endgroup$
    – toughkip
    Aug 2, 2022 at 7:03
  • $\begingroup$ The weights are distributed thanks to hyperparameters (mainly window size and learning rate) and the good results probably come from many trials and errors, like many DL models. If you want to know how it works precisely step by step, you can look this notebook. github.com/chiaminchuang/A-Neural-Probabilistic-Language-Model/… $\endgroup$
    – Nicolas Martin
    Aug 2, 2022 at 14:07
  • $\begingroup$ Does it answer your question? Please let me know if you need more details. $\endgroup$
    – Nicolas Martin
    Aug 3, 2022 at 15:32
  • 1
    $\begingroup$ This doesn't answer my question, unfortunately. I'm still gathering information and reading a few papers such as the spiritual successors to the NNLM above (Word2Vec, Glove, ..). I assume to find relevant information about my question there. $\endgroup$
    – toughkip
    Aug 5, 2022 at 6:39
  • $\begingroup$ Sorry for that, and please let me know if you find any relevant answers to your question. Although a mathematical understanding might be possible, many DL topics start with a conceptual idea that is improved through trials and errors, rather than rigorous logic. I might be wrong for NNLM, but the best way to understand such an algorithm is to ask authors themselves or redo it. $\endgroup$
    – Nicolas Martin
    Aug 5, 2022 at 7:31
0
$\begingroup$

The process of learning word vectors is :

  • Find the neighboring words
  • Vectors of neighbouring words is input to NN. Output is word vector of target

Take an example : The word vectors are learned from two sentences -

  • Address is 734, 3rd street, London.
  • The location is 734,3rd street London.

NN Learns that both Address, Location occurs near to London, street. They both share similar vectors

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.