In NLP word embedding represent word as number but after reading many blog i found that word are represent as vectors ? so what is word embedding exactly and Why words are represented in vector and the vector size is huge. what values dose that vector represent ?
what is the difference between word and character embedding ? please explain in simple term with simple example.
Most problems in NLP require the system to understand the semantic meaning of the text and not just the arrangement of specific words.
Semantic understanding enables a system to say that, "I am happy" and "It's joyful", have the same meaning.
To incorporate this feature to a system, we present words of a particular language in form of vectors. Often called as embeddings, they help in establishing similarities between words and phrases.
For instance, a vector representing the word "happy" will lie in the vicinity of the vectors representing the words "joy", "pleasure", "sad" etc. These vectors are high dimensional but using PCA or other dimensionality reduction techniques they are brought down to 3 dimensions where they could be visualized.
That's why we encode words in the form of vectors. We often use cosine similarity to determine the closest vector to a given vector in analysing sematic similarity.
For an intuition, the 3D space which contains vectors for all possible English words could be thought of our knowledge base. We tend to keep similar words together in our mind. If we are talking about fast food, for instance, our brain would capture the region of the knowledge base to retreive words related with fast food like "burgers", "chicken" etc.
There are several types of vector representations for words and characters, I'm assuming here the primary interest is dense representations that are used commonly in deep learning today.
The purpose of word or character vectorization was to transform sentences or documents into feature sets that could be used for machine learning. In traditional NLP, several methods were used for vectorization such as count vectorization and TF-IDF vectorization. These methods essentially would produce matrices based on the occurrence of words in documents. For instance, count vectorization of a set of documents would return a matrix with size n_documents X n_words (rows of documents, columns of wrods). Each column would be a vector representation of a word. Considering that there are a huge number of words in languages and only a few words would appear frequently, these vectorization matrices would be enormous, but contain mostly zeros i.e. sparse representations. There are a few problems with this approach:
Because of these problems, researchers were motivated to find better representations.
The two seminal works on dense representations were Word2Vec and GloVe. These methods produced dense, fixed-length vector representations of words rather than being dependent on the number of training samples, which solved problems 1 and 2 (from above). To solve problem 3, these models were trained on very, very large corpora (see section 4.2. of GloVe) [2].
The result from these methods (which was a surprising results) was that the dense vectors appear to contain some amount of semantic information i.e. in some sense word meaning was embedded into these vectors. Consider the following quote from the Word2Vec paper:
"Somewhat surprisingly, these questions can be answered by performing simple algebraic operations with the vector representation of words. To find a word that is similar to small in the same sense as biggest is similar to big, we can simply compute vector X = vector(”biggest”)−vector(”big”) + vector(”small”)... When the word vectors are well trained, it is possible to find the correct answer (word smallest) using this method." [1]
To answer the first question (what are these representations and why are vector used):
These dense word vectors efficiently represent (in some sense) words in terms of semantics.
Ok so where does the vector length come from? For Word2Vec and GloVe, typically a size of 300 is used/recommended. This comes from the GloVe paper. The following image [2] from the GloVe paper shows the performance of GloVe vectors in semantic and syntactic tasks. The performance essentially flattens out at a vector size of 300.
These dense representations has nice properties and have some significant advantages over traditional methods, but they still require a large embedding matrix. The GloVe paper mentions that only the 400,000 most common words out of several billion were used. So scale is still an issue. Also, when only a small subset of words are selected for embedding a large number may be "outside the vocabulary" (OOV). These words all get mapped to a single OOV vector. Additionally, there are other issues like misspellings. This motivated embeddings of characters and subwords.
As deep learning in NLP exploded, larger and larger vocabulary sizes where needed. Character and subword embeddings were an attempt to limit the size of embedding matrices such as in BERT. However, these types of embeddings do not encode the same deep sematics that word embeddings encode.
Character embeddings are constructed in similar fashion to the way that word embeddings are constructed. However, instead of embedding at the word level, the vectors represent each character in a language. For example, instead a vector for "king", there would be a separate vector for each of the letters: "k", "i", "n", and "g". As mentioned these types of embeddings do not encode the same type of information that word embeddings contain. Instead, character level embedding can be thought of encoded lexical information and may be used to enhance or enrich word level emebddings (see Enriching Word Vectors with Subword Information). While some research on use of character embeddings has been done (see [3]), character level embeddings are generally shallow in meaning.
As mentioned in [3] character-level embeddings have some advantages over word level embeddings such as
[1] Efficient Estimation of Word Representations in Vector Space
[2] GloVe: Global Vectors for Word Representation
[3] Character-level Convolutional Networks for Text Classification
First think of it like this. The most naive way you would encode words such that you can put it to neural network model is one hot encoding. If this is the case you will notice that your encoding vector size will grow linearly with your vocabulary size and on top of that it is sparse(without proper handling it is inefficient).
Second, your encoding doesn't have any "meaning" and you are basically asking the network to do the heavy lifting on figuring out meanings behind which could work but not ideal.
So how can we help our network? Humans have prior knowledge of how languages works and we try to instill them in the form of embeddings. Based on this we helped our neural network by giving a simpler task of "connecting the dots". Some other features that we want is we can have a measure semantic similarity. Which I believe is mentioned by Shubam.
To wrap up we solved 2 problems. First, in some sense we are doing dimensionality reduction. Second, we are able to project words in our vocabulary in a more meaningful way.
You can apply similar ideas for character, but probably make less sense.
