In NLP word embedding represent word as number but after reading many blog i found that word are represent as vectors ? so what is word embedding exactly and Why words are represented in vector and the vector size is huge. what values dose that vector represent ?

what is the difference between word and character embedding ? please explain in simple term with simple example.


Most problems in NLP require the system to understand the semantic meaning of the text and not just the arrangement of specific words.

Semantic understanding enables a system to say that, "I am happy" and "It's joyful", have the same meaning.

To incorporate this feature to a system, we present words of a particular language in form of vectors. Often called as embeddings, they help in establishing similarities between words and phrases.

For instance, a vector representing the word "happy" will lie in the vicinity of the vectors representing the words "joy", "pleasure", "sad" etc. These vectors are high dimensional but using PCA or other dimensionality reduction techniques they are brought down to 3 dimensions where they could be visualized.

That's why we encode words in the form of vectors. We often use cosine similarity to determine the closest vector to a given vector in analysing sematic similarity.

For an intuition, the 3D space which contains vectors for all possible English words could be thought of our knowledge base. We tend to keep similar words together in our mind. If we are talking about fast food, for instance, our brain would capture the region of the knowledge base to retreive words related with fast food like "burgers", "chicken" etc.

| improve this answer | |
  • $\begingroup$ Thank you @Shubham Panchal .Say for example the word "happy" is represented as vector with 3dimensions/values[0.01,0.02,-0.01] Assume for the sake of my understanding 0.01 represent distance between happy and joy. 0.02 represent distance between happy and pleasure and -0.01 represent distance between happy and sad (keeping minus as its opposite ) . Similar the vectors for joy,pleasure,sad have same values .Is my understanding correct ? Is there any ways to visualize the actual words represented by values in a particular word vector values like how i have shown it . $\endgroup$ – Aj_MLstater Oct 9 '19 at 14:42
  • $\begingroup$ Yes, exactly. We may consider the euclidean distance between them $\endgroup$ – Shubham Panchal Oct 10 '19 at 0:29
  • $\begingroup$ thank you could you please also explain in simple terms about character embedding as well ? $\endgroup$ – Aj_MLstater Oct 10 '19 at 5:41

First think of it like this. The most naive way you would encode words such that you can put it to neural network model is one hot encoding. If this is the case you will notice that your encoding vector size will grow linearly with your vocabulary size and on top of that it is sparse(without proper handling it is inefficient).

Second, your encoding doesn't have any "meaning" and you are basically asking the network to do the heavy lifting on figuring out meanings behind which could work but not ideal.

So how can we help our network? Humans have prior knowledge of how languages works and we try to instill them in the form of embeddings. Based on this we helped our neural network by giving a simpler task of "connecting the dots". Some other features that we want is we can have a measure semantic similarity. Which I believe is mentioned by Shubam.

To wrap up we solved 2 problems. First, in some sense we are doing dimensionality reduction. Second, we are able to project words in our vocabulary in a more meaningful way.

You can apply similar ideas for character, but probably make less sense.

| improve this answer | |
  • $\begingroup$ Hi Yohanes, thanks for your reply Can you explain it with an example ? $\endgroup$ – Aj_MLstater Nov 11 '19 at 11:11
  • $\begingroup$ Hi each embeddings "model" language differently, fasttext and glove which is simple word2vec model(substitute word to vector) or more complicated embedding like in ELMO and more modern nlp model. You might need to go to read through each one of them to get clearer idea what they are achieving. But the general intuition is still that we are instilling our knowledge of language model in the form of these embedding and the proceeding layers will only need to connect these dots $\endgroup$ – Yohanes Alfredo Nov 11 '19 at 11:33
  • $\begingroup$ I could probably put some example related to what I am explaining above. Lets say we have a vocabulary list of 10k words as if we use one hot encoding we will have this vector of size 10k to input to our network. But what if we have 100k words, each word wll be represented as vector of size 100k, if this is not bad enough, think what will happen if we have dense layer next, we will have weights of size (100000, number of hidden neuron) which will be memory inefficient. That is one layer but what if there are more. $\endgroup$ – Yohanes Alfredo Nov 11 '19 at 11:42
  • $\begingroup$ For the second part say I have three words in my vocab, [cry,boy,sad]. Now one hot encoding of this are [1,0,0], [0,1,0], and [0,0,1] respectively. Does this encoding scheme conveys any meaning? within euclidean space cry and boy is as close as sad. Simple Neural network might eventually discover the relationship when we train our model for our task, but it will be very inefficient and might not perform well. $\endgroup$ – Yohanes Alfredo Nov 11 '19 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.