# Tag Info

15

One-Hot Encoding is a general method that can vectorize any categorical features. It is simple and fast to create and update the vectorization, just add a new entry in the vector with a one for each new category. However, that speed and simplicity also leads to the "curse of dimensionality" by creating a new dimension for each category. Embedding ...

14

For those who are interested, I've spent some time, finally figured out that the problem was the way one has to prepare the categorical encoding for the Entity Embedding suitable for a neural network architecture; unfortunately none of the examples provided in blogposts or Kaggle kernels were clear about this step! Here is the link to the repository ...

12

Actually they are 3 different things (embedding layer, word2vec, autoencoder), though they can be used to solve similar problems. (i.e. dense representation of data) Autoencoder is a type of neural network where the inputs and outputs are the same but in the hidden layer the dimensionality is reduced in order to get a more dense representation of the data. ...

8

Yep, you're right! As you know, it's difficult for machine learning models to use natural language directly, so it helps to transform words into some meaningful numeric representation. This process is called word embedding, and finding word embeddings is the task of the keras Embedding layer. Ideally, word embeddings will be semantically meaningful, so ...

5

Let us try and understand how Word2Vector actually works before looking at distances: There are 2 ways of generating vectors for a word : Continuous bag of words Skip grams The following diagram explains the difference between the two approaches. In case you want to further understand the nitty gritty of these two approaches, there are tons of blogs out ...

5

It seems that Embedding vector is the best solution here. However, you may consider a variant of the one-hot encoding called 'one-hot hashing trick". In this variant, when the number of unique words is too large to be assigned a unique index in a dictionary, one may hash words of into vector of fixed size. One advantage in your use case is that you may ...

4

Leland's answer is exactly correct regarding why an autoencoder wouldn't be useful. Let me expand upon that point: Autoencoders and other dimensionality reduction techniques attempt to keep objects that are "close" together in your high dimensional space also close in the lower dimensional space. Often the measure of closeness the autoencoder learns leads ...

4

It might be useful to think of this in terms of orthogonality. You state that "categories are not correlating in any way", which effectively means each category should be completely orthogonal to all the others -- otherwise one would be representing some degree of correlation (or anti-correlation) between at least two categories (to the degree that those ...

4

Calling StarSpace a neural model would be misleading I think. You could certainly think of the it as a neural network with a single layer and a linear activation function, but I think don't think that would be very illuminating. They didn't discuss the architecture much in that paper for a reason- there isn't really any in terms of layers of neurons, ...

4

An embedding layer is in fact a linear layer. It maps the input, using a matrix multiplication, to the output, without any activation function after the multiplication. Therefore, the backpropagation is exactly as you would do with linear layer. Why don't we just call it linear layer, then? At theory level, an embedding layer performs a matrix ...

4

I think post below is a good resource. https://developers.googleblog.com/2017/11/introducing-tensorflow-feature-columns.html Basically, all categorical variable is initially converted to one-hot encoding, then layer size defined by dimension argument is stacked on top of one-hot encoding; thus learning optimal representation of categorical variable based ...

3

Use tf.gather(). Single instance case In the example below, we selected a variable number of embedding vectors from the matrix embedding. The selection indexing vector user can be of variable length. Then we calculate the average embedding. with tf.Graph().as_default(): embedding = tf.placeholder(shape=[10,3], dtype=tf.float32) user = tf....

3

There is no problem with the fact that your regularization loss is going up. The cost function of your model is a weighted sum of the regularization loss and the base loss, so during training the model looks to minimize them together, but eventually it comes to a point where it has to choose and minimize one on the expense of the other. The fact that it is ...

3

The suggestion of ncasas is a good one but not very clean. This ordering makes a lot of sense when it's 1-dimensional, but when you introduce more features the ordering will become more and more arbitrary. This is a problem I have come accross multiple times. This paper (https://arxiv.org/pdf/1612.04530.pdf) tries to tackle permutation equivariance, which is ...

3

There are several ways you can obtain document embeddings. If you want to obtain a vector of a document that is not part of the trained doc2vec model, gensim provides a method called infer_vector which allows to you map embeddings. You can also use bert-as-service to generate sentence level embeddings. I would recommend using Google's Universal Sentence ...

3

You are absolutely right about vocabulary size. I am actually conducting research on making character-level more effective. Here is why word-level tokens are often favoured despite, characters requiring a much smaller vocabulary size. Bag-of-word In a bag-of-words scenario, it is pretty obvious. First, the name. Second, if you receive a word cloud of most ...

2

No it is not possible to preserve relative distances when reducing dimensions for arbitrary data. This is not due to a property of auto-encoders compared to e.g. PCA or T-SNE. It is due to geometry. You can see this relatively easily by considering a reduction of dimensions from 3 to 2, and examining a tetrahedron where all four corner points are 1 unit ...

2

An embedding is a mapping from discrete objects, such as words, to vectors of real numbers. - Tensorflow/Embeddings With reference to the FaceNet paper, I should say that embedding here simply means the tensor obtained by performing a forward propagation over an image. The obtained embedding of the image and the target image are then compared to find the ...

2

Assume you have features wich lie in a $R^n$ space, e.g. your input is a picture with $28 \times 28$ pixels, then $n$ would be $28 \times 28 = 784$. Now you can "embedd" your features into another $R^d$ space, where often $d < n$. This way you learn a rich representation of your input. When you compress your $784$ input-pixels to, lets say $64$ you ...

2

If you mean completely new set of features for prediction: it will not be helpful. Your model 'learns' something on the training feature space and you hope to apply the learning to new data-points in the same feature space. If the feature space for prediction is completely new, the learning will be useless! If you mean somewhat new set of features: that ...

2

It's called a prefix tree. And because the ICD codes are human made, and humans tend to think in categories, there is indeed such a structure in this codes. It's just that humans first thought of the structure, then made these codes to reflect that structure, not the other way round. What you are now doing is reverse-engineering this...

2

Most problems in NLP require the system to understand the semantic meaning of the text and not just the arrangement of specific words. Semantic understanding enables a system to say that, "I am happy" and "It's joyful", have the same meaning. To incorporate this feature to a system, we present words of a particular language in form of vectors. Often ...

2

If the sequence of watches is meaningful, then you do need some kind of classifier that creates user/item embeddings from the sequence of watches, so you're probably looking at GRUs and LSTMs. Neural nets aren't overkill here per se. This is a pretty good treatment of the topic: https://towardsdatascience.com/introduction-to-recommender-system-part-2-...

2

One possibility would be to represent the zip codes using some transformation that could be applied to new (unseen) zip codes as well. For example, could you re-represent zip codes as latitude + longitude?

2

There is a theoretical lower bound for embedding dimension I would urge you to read this paper, but the gist of it is dimension could be chosen based on corpus statistics GLOVE paper discussed embedding, check page 7 for graphs. What I want to say with this reference is that you can treat it as hyperparameter and find your optimal value. EDIT: Here is my ...

2

I got it. You define a new model, which has an input, the shared embedding layer and a flattened output. Pass the output of .predict() from that model to y parameter of your main model's .fit() call, in like fashion: NUMERIC_FEATURES = [ # Define the subset of features that need passing to the numeric input layer ] vocab_size = 10000 # number of items ...

2

Don't know if this is what you need but I know of the Ampligraph library: Python library for Representation Learning on Knowledge Graphs

2

There are libraries that are specialized in exactly that task, for instance FAISS by Facebook AI Research: Faiss is a library for efficient similarity search and clustering of dense vectors. It contains algorithms that search in sets of vectors of any size, up to ones that possibly do not fit in RAM. It also contains supporting code for evaluation and ...

2

Upon further research I found that during training forward language models try to predict the next word in a sequence. Backwards language models on the other hand start at the end of a sequence and attempt to predict the proceeding word. It seems that by stacking both forward and backwards models produced from the same data-set you get better results than ...

2

So, you want to make embeddings for sequences of items, and order doesn't make much sense. You don't have a specific objective, just want to retrieve embeddings with some conventional properties, such as natural clustering. If you can train word2vec and take mean of item embeddings in a cart, you'll probably get noisy vectors. It doesn't work as well as ...

Only top voted, non community-wiki answers of a minimum length are eligible