What does embedding mean in machine learning?

Question

I just met a terminology called "embedding" in a paper regarding deep learning. The context is "multi-modal embedding"

My guess: embedding of something is extract some feature of sth,to form a vector.

I couldn't get the explicit meaning for this terminology and that stops me from fully understanding the author's idea and model mechanism

I check the dictionary and search on line,but the explanation is based more on the real life meaning rather than meaning as a machine learning terminology.

And that raise a more generalized and frequently met question, when you find some machine learning terminology/word that you can't understand well, where can you get the solution, some specific way to google? join a machine learning group? raise a question in stack exchange?

Answer 1

In the context of machine learning, an embedding is a low-dimensional, learned continuous vector representation of discrete variables into which you can translate high-dimensional vectors. Generally, embeddings make ML models more efficient and easier to work with, and can be used with other models as well.

Typically, when I stumble upon jargon I'm not familiar with I first turn to Google, and if it can't be found I ping my colleagues and data science forums.

Answer 2

According to all answers(Thank you) and my google search I got a better understanding, So my newly updated understanding is:

The embedding in machine learning or NLP is actually a technique mapping from words to vectors which you can do better analysis or relating, for example, "toyota" or "honda" can be hardly related in words, but in vector space it can be set to very close according to some measure, also you can strengthen the relation ship of word by setting: king-man+woman = Queen.

so we can set boy to (1,0) and then set girl to (-1,0) to show they are in the same dimension but the meaning is just opposite. And all nouns that just diff in gender can be parallel~

My initial guess that embedding is extracting features from something is close but not specific enough.

And for my last point when you met a jargon in some special area how to quickly get the essential meaning of it, I still didn't find a very good way, maybe a website that can explain the meaning of jargon in that area will save great time for us.

Answer 3

For me embedding is used to represent big sparse matrix into smaller dimensions, where each dimension(feature) represent a meaningful association with other elements in the embedding matrix.

Consider an example of NLP. Where each sentence broken down into words(also called token). Such set of different words make a vocabulary for NLP. Generally vocabulary have millions of words. All such words can be uniquely represented as OneHotEncoding.

Demerits of OneHotEncoding representation of words:

1. In case of large vocabulary, OneHotEncoding representation needs a big chunk of memory and computationally become very expensive.
2. OneHotEncoding is used to represent categorical values, where each entity is independent to other one, whereas words in vocabulary represent some association in terms of similar meanings or in some other way. OneHotEncoding not utilizing that capability for NLP.

In order to overcome both the issues, we use word Embedding, where each word represented in lesser dimension, where each dimension represent some sort of features and hence each dimension will have some values.

Answer 4

The LSA community seems to have first used the word “embedding” in Landauer et al. (1997), in a variant of its mathematical meaning as a mapping from one space or mathematical structure to another. In LSA, the word embedding seems to have described the mapping from the space of sparse count vectors to the latent space of SVD dense vectors. Although the word thus originally meant the mapping from one space to another, it has metonymically shifted to mean the resulting dense vector in the latent space. and it is in this sense that we currently use the word.

For the more generalized and frequently met questions, I'd like to recommend you read textbooks, especially classic ones.

Refrence:
Speech and Language Processing: An introduction to natural language processing

Answer 5

Embeddings are vector representations of a particular word.

In Machine learning, textual content has to be converted to numerical data to feed it into Algorithm.

One method is one hot encoding but it breaks down when we have large no of vocabulary. The size of word representation grows as the vocabulary grows. Also, it is sparse.

With embedding (fixed size vectors with lower dimension), the size of word representation can be controlled. Also, the vector representation stores the semantic relationship b/w words. There are pretrained embeddings Word2Vec, Glove etc available which can be used just as a lookup. Embeddings improve the performance of ML model significantly.

Answer 6

As well known, machine only identify 0 and 1. Therefore, we, for an instance, "encode" characters and symbols with ASCII codes. 0 & 1 can only code two characters. To make sure the code is unique for all characters, we have to use a series number to code them. The hard ware experienced 8, 16, 32, and 64 bits (now 64 bits is most popular).

Similarly, in my understanding, embedding is to convert non-digital inputs in a preset order, like words, videos, images, etc. into vectors. These vectors can be used to machine learning easily. The dimension of metrics depends on the features of objects. Few features would result in low dimension, vice versa.

For an instance, we would recommend movies to users. If users have only one feature, let's say UserID, and movies only has one feature, movie name. We can build up a matrix with UserID as row, and movie name as columns. If a user watched some movies, those intersection would be 1, otherwise would be 0.

       | movie1 | movie2 | movie3 | movie 4 | ... | movie n |
user1  |   1    |   0    |   0    |    1    | ... |    0    |
user2  |   0    |   0    |   0    |    0    | ... |    1    |
 ...   |  ...   |  ...   |  ...   |   ...   | ... |   ...   |
userm  |   1    |   1    |   0    |    0    | ... |    0    |

Above is a m X n matrix. Each row represents one user. Thus, for user1, we have a vector (1 0 0 1 ... 0), user2 (0 0 0 0 ... 1), similarly for the rest of users. If the movie has only name without any further features, it is hard for machine to learn and train. Now, if we arrange the movies from left to right based on the suitable age group, we then can recommend movie2 and 3 to user1.

Based on above example, we can classify the movies in different features (let's say t features), like animated cartoon, action movie, war movie, horror movie, etc. We can build a m X n X t matrix. Based on the watch history, machine can classify a user's watched movies into different group and recommend similar movies.