Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
2
votes
1answer
18 views

Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)? [closed]

Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors? If so, say for example we have a sentence and we pass it through the encoder to ...
0
votes
1answer
23 views

How to choose dimension of Keras embedding layer?

Looking for some guidelines to choose dimension of Keras word embedding layer. For example in a simplified movie review classification code: ...
1
vote
0answers
14 views

node2vec initial embedding

I am referring to the following implementation of node2vec: https://github.com/eliorc/node2vec Is there a way to have a "smart initialization" with node2vec, i.e., to start the algorithm ...
1
vote
2answers
35 views

Difference between CBOW and SKIP Gram word vectors

I have gone through several links but was not able to understand how CBOW and Skip Gram is trained from scratch? Any good link/blogs or books would be very helpful. ...
2
votes
1answer
26 views

Can we combine multiple K-Means Models as a single model?

I have a NLP problem statement where I use a Word2Vec embedding pre-trained model to convert key text to vectors and then on a set of terms run k-means clustering to get a final model for certain <...
1
vote
0answers
10 views

Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?

I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the ...
1
vote
0answers
15 views

Is the magnitude of a word vector correlated with the frequency of the word in a text?

In order to find the similarity between words, cosine similarity seems to be the most common measure to use. In a conversation that I had about this topic, someone mentioned that words that mean more ...
0
votes
0answers
5 views

Embed Sequence API in Tensorflow

My current tensorflow version is 2.1 and I'm using low level(core) tensorflow api's. Please provide a substitute of "tf.contrib.layers.embed_sequence". I have explored a lot, but could not ...
1
vote
0answers
27 views

Learning word embeddings by first learning character embeddings

I was going through various papers for NLU applications(Natural Language Understanding). There I have observed a common pattern that for a word embeddings, following 3 combinations are used (may be ...
2
votes
1answer
37 views

What's the best way to detect bible verse mentions in a text?

I have a set of 10 verses from the Bible in English. I want to detect the occurrence of any of these verses in a text. What would be the best way to go about doing this? Note that verses of the Bible ...
1
vote
1answer
18 views

Construct word2vec (CBOW) training data from beginning of sentence

When constructing training data for CBOW, Mikolov et al. suggest using the word from the center of a context window. What is the "best" approach to capturing words at the beginning/end of a ...
1
vote
1answer
37 views

Why is the cosine distance used to measure the similatiry between word embeddings?

While computing the similarity between the words, cosine similarity or distance is computed on word vectors. Why aren't other distance metrics such as Euclidean ...
0
votes
0answers
8 views

Size of data required to build instruction Embeddings

I am using different sizes of ARMv7 data to build instructions embedding using a conventional word2vec model which would output a numpy array of Embeddings that ...
0
votes
0answers
23 views

Does it make sense to use TF-IDF matrix as Embedding layer weights in Keras

Keras Embedding layer accept weights pre-trained from GloVe for example. A classic NN should look like this: ...
0
votes
0answers
9 views

Building a Keras text embedding model with cosine proximity

I am trying to build a word embedding keras model wherein I give as input a text that is converted to its corresponding input ids and masks (like input to an Albert model) and it gives me back a 768 ...
0
votes
0answers
14 views

Distance to the Center of Vocabulary of Word Embeddings

Suppose I: Generate a set of word embeddings (using word2vec or similar) based on a large but specific corpus Compute the centroid of all the words in the set Find the word(s) with the smallest (say ...
2
votes
1answer
25 views

What GPU size do I need to fine tune BERT base cased?

I want to fine tune BERT Multilingual but I'm not aware about the GPU requirements to train BERT Multilingual. I have GTX 1050ti 4GB on my local machine. I want to know what size of GPU is needed and ...
1
vote
1answer
60 views

Can we use sentence transformers to embed sentences without labels?

I was trying to use this project : https://github.com/UKPLab/sentence-transformers for embedding non english sentences, the language is not a human speaking language, its machine language (x86) but ...
1
vote
2answers
141 views

How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

First of all, I want to say that I am asking this question because I am interested in using BERT embeddings as document features to do clustering. I am using Transformers from the Hugging Face library....
0
votes
1answer
28 views

Is adding the embedded words of a sentence to represent the sentence a good approach?

I have a dataset of sentences in a non english language like : word1 word2 word3 word62 word5 word1 word2 Now i want to turn each variable length sentence to a fixed size vector to give it to my ...
0
votes
1answer
27 views

best approach to embed random length sequences of words as a fixed size vector without having a maximum length? [closed]

I have a dataset of sentences in a non-English language like: word1 word2 word3 word62 word5 word1 word2 and the length of each sentence is not fixed. Now, I want to represent each sentence as a ...
0
votes
0answers
28 views

Doc2Vec most_similar() returns cosine similarity for inferred out-of-training-set vectors

According to the gensim documentation the doc2vec function most_similar() can be called with a doctag and returns the top_n most similar documents (which were also present in the training set) Out-of-...
2
votes
2answers
36 views

Should I keep common stop-words when preprocessing for word embedding?

If I want to construct a word embedding by predicting a target word given context words, is it better to remove stop words or keep them? the quick brown fox jumped over the lazy dog or quick brown ...
0
votes
1answer
24 views

How keras.layers.embedding learn word embeddings?

I was trying some tensorflow tutorials and see that in all of them they use layers.embedding to learn these word embeddings, but how are these learned? , with a NN? which arquitecture? , or word2vec? ...
0
votes
1answer
27 views

how to train custom word2vec embeddings to find related articles?

I am beginner in machine learning. My project is to make search engine based on AI which shows related articles when we search on website. For this i decided to train my own embedding. I found two ...
1
vote
2answers
248 views

Can we use BERT for only word embedding and then use SVM/RNN to do intent classification?

According to this article, "Systems used for intent classification contain the following two components: Word embedding, and a classifier." This article also evaluated BERT+SVM and Word2Vec+...
1
vote
1answer
63 views

Understanding Transfer Learning of Word Embeddings

I can't quite visualize how transfer learning of pre-trained word embeddings is useful in an NLP task( say named entity recognition ) . I'm studying Andrew NG's Sequence Models course and he seems to ...
1
vote
1answer
24 views

word2vec: usefulness of context vectors in classification

I've been working on a NN-based classification system that accepts document vectors as input. I can't really talk about what I'm specifically training the neural net on, so i'm hoping for a more ...
2
votes
1answer
113 views

How do I read the cord_19_embeddings_2020-07-16.csv from the COVID-19 Open Research Dataset Challenge (CORD-19) on Kaggle?

What I'm trying to do I wanted to use the CORD19 word embeddings csv to map them to certain findings from the rest of the dataset, but as we can see there are no stings in the first column. The way I ...
1
vote
0answers
20 views

How best to embed large and noisy documents

I have a large corpus of documents (web pages) collected from various sites of around 10k-30k chars each, I am processing them to extract relevant text as much as possible, but they are never perfect. ...
0
votes
1answer
25 views

A way to init sentence embedding for unsupervised text clustering, better than glove wordvec?

For unsupervised text clustering, the key thing is the init embedding for text. If we want to use deepcluster for text, the problem for text is how to get the init embedding from deep model. BERT can ...
0
votes
0answers
7 views

how to make CBOW model suitable for only 10 outpus?

I want to modify CBOW model to output only 10 classes. my question is how to make this model perform better even if the number of output is much less than the number of vocabulary output? should I use ...
0
votes
1answer
23 views

Keep word2vexc/fasttext model loaded in memory without using API

I have to use Fasttext model to return word embeddings. In test I was calling it through API. Since there are too many words to compute embeddings, API call seems to be expensive. I would like to use ...
1
vote
0answers
16 views

How to do embedding for nested dictionary with varying size?

I'm working on an RL task in which the agent needs have some observation. Instead using images, I want to use available information of the environment as the observation. The information regarding the ...
1
vote
0answers
37 views

Literature on selecting specific dimensions in a word embedding vector

I am aware that the different dimensions in the word embedding represents different information and algebraic operations can be performed between two embeddings for example. Can anyone point me to ...
1
vote
1answer
19 views

GPU vs TPU for convolutional neural networks (NLP)

I am testing ideas on IMDB sentiment analysis task by using word embeddings + CNN approach. What could explain a significant difference in computation time in favor of GPU (~9 seconds per epoch) ...
3
votes
1answer
24 views

How to work with different Encoding for Foreign Languages

I've got a Word Embedding File called model.txt. This contains 100 Dimensional vectors for over a million French words. These words contain accented characters such ...
0
votes
0answers
8 views

what kind of distribution is followed by word and sentence vectors generated by TFIDF ,word2vec,glove,bert,flair?

what kind of distribution is followed by word or sentence embedding vectors generated by TFID or pretrained models like word2vec,glove,bert,flair ? is it continuous or discrete or any other ...
0
votes
0answers
13 views

Extract average embedding for class labels

I am training a neural net to predict class labels for documents. The first layer in my network is an embedding layer. Once the network is fully trained, I would like to extract an a single embedding ...
0
votes
1answer
77 views

Embedding of list of objects

I have a dataset where each sample is a list of ordered items, lets say grocery list , and a label from 6 categories . each list can have up to 120 items but the mean items is 12 items in a list. i ...
1
vote
0answers
13 views

Embedding layers trained on Amazon Reviews

I am working on research to perform sentiment analysis on Amazon reviews. My data is not labelled so I am now using Lexicon based sentiment analysis such as Vader. I am wondering if it is possible to ...
0
votes
1answer
59 views
0
votes
1answer
26 views

ELMo - How does the model transfer its learning/weights on new sentences

Word2vec and Glove embeddings have the same vector representation for every word in the corpus and does not take context into consideration. For eg: The dog does bark at people The bark of the tree ...
1
vote
0answers
43 views

Extracting vectors of FastText own model to use it on a NN

I have trained my own model of fasttext using the pretrained model of English available on their website with the next code: ...
4
votes
1answer
30 views

Why are character level models considered less effective than word level models?

I have read that character level models need more computation power than word embeddings, and this is one of the major reasons for their less effectiveness, but i got curious because the word ...
0
votes
0answers
41 views

Using word embeddings for kaggle?

Not sure, if this is the right forum so redirect me if it wrong. I have started on an NLP problem in kaggle. There i have word embeddings from google news, wiki, glove in a zipped folder. I want to ...
0
votes
0answers
16 views

Skip gram model on multiple sequence

From different examples, I have seen that getting an embedding from skip-gram model on a single sequence or a single corpus. However, if I have multiple sequences of same word or phases. How can I use ...
0
votes
0answers
13 views

Is the number of bidirectional LSTMs in encoder-decoder model equal to the maximum length of input text/characters?

I'm confused about this aspect of RNNs while trying to learn how seq2seq encoder-decoder works at https://machinelearningmastery.com/configure-encoder-decoder-model-neural-machine-translation/. It ...
0
votes
0answers
9 views

how many spectogram frames per input character does text-to-speech (TTS) system Tacotron-2 generate?

I've been reading on Tacotron-2, a text-to-speech system, that generates speech just-like humans (indistinguisahble from humans) using the github https://github.com/Rayhane-mamah/Tacotron-2. I'm very ...
0
votes
0answers
32 views

keras how to subset input in Model

I have a data of the following format: ...

1
2 3 4 5
7