# What is word embedding and character embedding ? Why words are represented in vector with huge size?

In NLP word embedding represent word as number but after reading many blog i found that word are represent as vectors ? so what is word embedding exactly and Why words are represented in vector and the vector size is huge. what values dose that vector represent ?

what is the difference between word and character embedding ? please explain in simple term with simple example.

Most problems in NLP require the system to understand the semantic meaning of the text and not just the arrangement of specific words.

Semantic understanding enables a system to say that, "I am happy" and "It's joyful", have the same meaning.

To incorporate this feature to a system, we present words of a particular language in form of vectors. Often called as embeddings, they help in establishing similarities between words and phrases.

For instance, a vector representing the word "happy" will lie in the vicinity of the vectors representing the words "joy", "pleasure", "sad" etc. These vectors are high dimensional but using PCA or other dimensionality reduction techniques they are brought down to 3 dimensions where they could be visualized.

That's why we encode words in the form of vectors. We often use cosine similarity to determine the closest vector to a given vector in analysing sematic similarity.

For an intuition, the 3D space which contains vectors for all possible English words could be thought of our knowledge base. We tend to keep similar words together in our mind. If we are talking about fast food, for instance, our brain would capture the region of the knowledge base to retreive words related with fast food like "burgers", "chicken" etc.

• Thank you @Shubham Panchal .Say for example the word "happy" is represented as vector with 3dimensions/values[0.01,0.02,-0.01] Assume for the sake of my understanding 0.01 represent distance between happy and joy. 0.02 represent distance between happy and pleasure and -0.01 represent distance between happy and sad (keeping minus as its opposite ) . Similar the vectors for joy,pleasure,sad have same values .Is my understanding correct ? Is there any ways to visualize the actual words represented by values in a particular word vector values like how i have shown it . – Aj_MLstater Oct 9 '19 at 14:42
• Yes, exactly. We may consider the euclidean distance between them – Shubham Panchal Oct 10 '19 at 0:29
• thank you could you please also explain in simple terms about character embedding as well ? – Aj_MLstater Oct 10 '19 at 5:41

First think of it like this. The most naive way you would encode words such that you can put it to neural network model is one hot encoding. If this is the case you will notice that your encoding vector size will grow linearly with your vocabulary size and on top of that it is sparse(without proper handling it is inefficient).

Second, your encoding doesn't have any "meaning" and you are basically asking the network to do the heavy lifting on figuring out meanings behind which could work but not ideal.

So how can we help our network? Humans have prior knowledge of how languages works and we try to instill them in the form of embeddings. Based on this we helped our neural network by giving a simpler task of "connecting the dots". Some other features that we want is we can have a measure semantic similarity. Which I believe is mentioned by Shubam.

To wrap up we solved 2 problems. First, in some sense we are doing dimensionality reduction. Second, we are able to project words in our vocabulary in a more meaningful way.

You can apply similar ideas for character, but probably make less sense.

• Hi Yohanes, thanks for your reply Can you explain it with an example ? – Aj_MLstater Nov 11 '19 at 11:11
• Hi each embeddings "model" language differently, fasttext and glove which is simple word2vec model(substitute word to vector) or more complicated embedding like in ELMO and more modern nlp model. You might need to go to read through each one of them to get clearer idea what they are achieving. But the general intuition is still that we are instilling our knowledge of language model in the form of these embedding and the proceeding layers will only need to connect these dots – Yohanes Alfredo Nov 11 '19 at 11:33
• I could probably put some example related to what I am explaining above. Lets say we have a vocabulary list of 10k words as if we use one hot encoding we will have this vector of size 10k to input to our network. But what if we have 100k words, each word wll be represented as vector of size 100k, if this is not bad enough, think what will happen if we have dense layer next, we will have weights of size (100000, number of hidden neuron) which will be memory inefficient. That is one layer but what if there are more. – Yohanes Alfredo Nov 11 '19 at 11:42
• For the second part say I have three words in my vocab, [cry,boy,sad]. Now one hot encoding of this are [1,0,0], [0,1,0], and [0,0,1] respectively. Does this encoding scheme conveys any meaning? within euclidean space cry and boy is as close as sad. Simple Neural network might eventually discover the relationship when we train our model for our task, but it will be very inefficient and might not perform well. – Yohanes Alfredo Nov 11 '19 at 11:48

There are several types of vector representations for words and characters, I'm assuming here the primary interest is dense representations that are used commonly in deep learning today.

# First, Some Background

The purpose of word or character vectorization was to transform sentences or documents into feature sets that could be used for machine learning. In traditional NLP, several methods were used for vectorization such as count vectorization and TF-IDF vectorization. These methods essentially would produce matrices based on the occurrence of words in documents. For instance, count vectorization of a set of documents would return a matrix with size n_documents X n_words (rows of documents, columns of wrods). Each column would be a vector representation of a word. Considering that there are a huge number of words in languages and only a few words would appear frequently, these vectorization matrices would be enormous, but contain mostly zeros i.e. sparse representations. There are a few problems with this approach:

1. Extremely high dimensional data doesn't work well in most ML methods i.e. curse of dimensionality.
2. These sparse representations are inefficient (particularly memory inefficient). Often needs to be compressed by some method such as PCA and data reduction such as removal of stop words.
3. These vectorization methods are dependent on the training samples and doesn't generalize well.

Because of these problems, researchers were motivated to find better representations.

# Dense Representations of Words

The two seminal works on dense representations were Word2Vec and GloVe. These methods produced dense, fixed-length vector representations of words rather than being dependent on the number of training samples, which solved problems 1 and 2 (from above). To solve problem 3, these models were trained on very, very large corpora (see section 4.2. of GloVe) [2].

The result from these methods (which was a surprising results) was that the dense vectors appear to contain some amount of semantic information i.e. in some sense word meaning was embedded into these vectors. Consider the following quote from the Word2Vec paper:

"Somewhat surprisingly, these questions can be answered by performing simple algebraic operations with the vector representation of words. To find a word that is similar to small in the same sense as biggest is similar to big, we can simply compute vector X = vector(”biggest”)−vector(”big”) + vector(”small”)... When the word vectors are well trained, it is possible to find the correct answer (word smallest) using this method." [1]

To answer the first question (what are these representations and why are vector used):

These dense word vectors efficiently represent (in some sense) words in terms of semantics.

Ok so where does the vector length come from? For Word2Vec and GloVe, typically a size of 300 is used/recommended. This comes from the GloVe paper. The following image [2] from the GloVe paper shows the performance of GloVe vectors in semantic and syntactic tasks. The performance essentially flattens out at a vector size of 300.

These dense representations has nice properties and have some significant advantages over traditional methods, but they still require a large embedding matrix. The GloVe paper mentions that only the 400,000 most common words out of several billion were used. So scale is still an issue. Also, when only a small subset of words are selected for embedding a large number may be "outside the vocabulary" (OOV). These words all get mapped to a single OOV vector. Additionally, there are other issues like misspellings. This motivated embeddings of characters and subwords.

# Character Embeddings

As deep learning in NLP exploded, larger and larger vocabulary sizes where needed. Character and subword embeddings were an attempt to limit the size of embedding matrices such as in BERT. However, these types of embeddings do not encode the same deep sematics that word embeddings encode.

Character embeddings are constructed in similar fashion to the way that word embeddings are constructed. However, instead of embedding at the word level, the vectors represent each character in a language. For example, instead a vector for "king", there would be a separate vector for each of the letters: "k", "i", "n", and "g". As mentioned these types of embeddings do not encode the same type of information that word embeddings contain. Instead, character level embedding can be thought of encoded lexical information and may be used to enhance or enrich word level emebddings (see Enriching Word Vectors with Subword Information). While some research on use of character embeddings has been done (see [3]), character level embeddings are generally shallow in meaning.

As mentioned in [3] character-level embeddings have some advantages over word level embeddings such as

• Able to handle new slang words and misspellings
• The required embedding matrix is much smaller than what is required for word level embeddings.

# References

[1] Efficient Estimation of Word Representations in Vector Space

[2] GloVe: Global Vectors for Word Representation

[3] Character-level Convolutional Networks for Text Classification