21 votes
Accepted

One Hot Encoding vs Word Embedding - When to choose one or another?

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each ...
user avatar
14 votes
Accepted

Confusion about Entity Embeddings of Categorical Variables - Working Example!

For those who are interested, I've spent some time, finally figured out that the problem was the way one has to prepare the categorical encoding for the Entity Embedding suitable for a neural network ...
user avatar
  • 3,936
12 votes

What is the difference between and Embedding Layer and an Autoencoder?

Actually they are 3 different things (embedding layer, word2vec, autoencoder), though they can be used to solve similar problems. (i.e. dense representation of data) Autoencoder is a type of neural ...
user avatar
  • 820
9 votes
Accepted

Difference between Gensim word2vec and keras Embedding layer

Yep, you're right! As you know, it's difficult for machine learning models to use natural language directly, so it helps to transform words into some meaningful numeric representation. This process ...
user avatar
  • 2,261
6 votes
Accepted

Backpropgating error to emedding matrix

An embedding layer is in fact a linear layer. It maps the input, using a matrix multiplication, to the output, without any activation function after the multiplication. Therefore, the backpropagation ...
user avatar
  • 5,594
6 votes

One Hot Encoding vs Word Embedding - When to choose one or another?

It seems that Embedding vector is the best solution here. However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick". In this variant, when the number of unique words ...
user avatar
  • 296
5 votes

word2vec word embeddings creates very distant vectors, closest cosine similarity is still very far, only 0.7

Let us try and understand how Word2Vector actually works before looking at distances: There are 2 ways of generating vectors for a word : Continuous bag of words Skip grams The following diagram ...
user avatar
4 votes

Auto-Encoder to condense (pre-process) large one-hot input vectors?

Leland's answer is exactly correct regarding why an autoencoder wouldn't be useful. Let me expand upon that point: Autoencoders and other dimensionality reduction techniques attempt to keep objects ...
user avatar
  • 2,341
4 votes
Accepted

Auto-Encoder to condense (pre-process) large one-hot input vectors?

It might be useful to think of this in terms of orthogonality. You state that "categories are not correlating in any way", which effectively means each category should be completely orthogonal to all ...
user avatar
4 votes
Accepted

What is the neural network architecture behind Facebook's Starspace model?

Calling StarSpace a neural model would be misleading I think. You could certainly think of the it as a neural network with a single layer and a linear activation function, but I think don't think that ...
user avatar
4 votes
Accepted

Choosing an embedding feature dimension

I think post below is a good resource. https://developers.googleblog.com/2017/11/introducing-tensorflow-feature-columns.html Basically, all categorical variable is initially converted to one-hot ...
user avatar
  • 191
3 votes
Accepted

Loss and Regularization inference

There is no problem with the fact that your regularization loss is going up. The cost function of your model is a weighted sum of the regularization loss and the base loss, so during training the ...
user avatar
  • 2,080
3 votes
Accepted

Tensorflow: how to look up and average a different amount of embedding vectors per training instance, with multiple training instances per minibatch?

Use tf.gather(). Single instance case In the example below, we selected a variable number of embedding vectors from the matrix ...
user avatar
  • 2,119
3 votes

Unordered Input

The suggestion of ncasas is a good one but not very clean. This ordering makes a lot of sense when it's 1-dimensional, but when you introduce more features the ordering will become more and more ...
user avatar
3 votes
Accepted

Use embeddings to find similarity between documents

There are several ways you can obtain document embeddings. If you want to obtain a vector of a document that is not part of the trained doc2vec model, gensim provides a method called ...
user avatar
  • 319
3 votes

Flair Embeddings - Significance of Backwards vs Forwards?

Upon further research I found that during training forward language models try to predict the next word in a sequence. Backwards language models on the other hand start at the end of a sequence and ...
user avatar
3 votes

Why are character level models considered less effective than word level models?

You are absolutely right about vocabulary size. I am actually conducting research on making character-level more effective. Here is why word-level tokens are often favoured despite, characters ...
user avatar
2 votes

What does the embedding mean in the FaceNet?

An embedding is a mapping from discrete objects, such as words, to vectors of real numbers. - Tensorflow/Embeddings With reference to the FaceNet paper, I should say that embedding here simply ...
user avatar
  • 2,225
2 votes
Accepted

What does the embedding mean in the FaceNet?

Assume you have features wich lie in a $R^n$ space, e.g. your input is a picture with $28 \times 28$ pixels, then $n$ would be $28 \times 28 = 784$. Now you can "embedd" your features into another $...
user avatar
2 votes
Accepted

What tasks you train with one set of features and predict with another?

If you mean completely new set of features for prediction: it will not be helpful. Your model 'learns' something on the training feature space and you hope to apply the learning to new data-points in ...
user avatar
  • 1,928
2 votes

Retain similarity distances when using an autoencoder for dimensionality reduction

No it is not possible to preserve relative distances when reducing dimensions for arbitrary data. This is not due to a property of auto-encoders compared to e.g. PCA or T-SNE. It is due to geometry. ...
user avatar
  • 27.2k
2 votes

Finding the best "depth" of ICD9 codes with pseudo-hierarchical clustering

It's called a prefix tree. And because the ICD codes are human made, and humans tend to think in categories, there is indeed such a structure in this codes. It's just that humans first thought of ...
user avatar
2 votes

What is word embedding and character embedding ? Why words are represented in vector with huge size?

Most problems in NLP require the system to understand the semantic meaning of the text and not just the arrangement of specific words. Semantic understanding enables a system to say that, "I am ...
user avatar
2 votes

What is word embedding and character embedding ? Why words are represented in vector with huge size?

There are several types of vector representations for words and characters, I'm assuming here the primary interest is dense representations that are used commonly in deep learning today. First, Some ...
user avatar
  • 21
2 votes

How to encode an array of categories to feed into sklearn

If the sequence of watches is meaningful, then you do need some kind of classifier that creates user/item embeddings from the sequence of watches, so you're probably looking at GRUs and LSTMs. Neural ...
user avatar
  • 6,386
2 votes
Accepted

How to handle different categorical embedding sizes in hold out data set

One possibility would be to represent the zip codes using some transformation that could be applied to new (unseen) zip codes as well. For example, could you re-represent zip codes as latitude + ...
user avatar
  • 119
2 votes

Choosing the size of Character Embedding for Language Generation models

There is a theoretical lower bound for embedding dimension I would urge you to read this paper, but the gist of it is dimension could be chosen based on corpus statistics GLOVE paper discussed ...
user avatar
  • 5,271
2 votes
Accepted

Using embedding layer output as input to .fit() call in Keras

I got it. You define a new model, which has an input, the shared embedding layer and a flattened output. Pass the output of .predict() from that model to ...
user avatar
  • 1,664
2 votes

Scalable way to train graph embeddings

Don't know if this is what you need but I know of the Ampligraph library: Python library for Representation Learning on Knowledge Graphs
user avatar
  • 20.8k
2 votes
Accepted

Fastest way for 1 vs all lookup on embeddings

There are libraries that are specialized in exactly that task, for instance FAISS by Facebook AI Research: Faiss is a library for efficient similarity search and clustering of dense vectors. It ...
user avatar
  • 15.1k

Only top scored, non community-wiki answers of a minimum length are eligible