Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

0
votes
1answer
31 views

Possible reasons for word2vec learning context words as most similar rather than words in similar contexts

I am observing my word2vec model learning context words as most similar rather than words in similar contexts. I don't understand why it (word2vec in general, not my model in particular) can behave ...
1
vote
1answer
19 views

How can I parallelize GloVe reverse lookups in PyTorch?

I feel like I'm missing something obvious here because I can't find any discussion of this. I want to do a lot of reverse lookups (nearest neighbor distance searches) on the GloVe embeddings for a ...
1
vote
1answer
23 views

Updating Google News Word2vec Word Embedding?

Is it possible to update the Google News Word Embedding with a custom text dataset (text data pertaining to a particular domain) ? Google News Word2Vec - Word Embedding clearly helps us to come with ...
1
vote
0answers
8 views

Why ELMo's word embedding can represent the word better than glove?

I have read the code of ELMo: https://github.com/allenai/bilm-tf Based on my understanding, ELMo first init an word embedding matrix A for all the word and then ...
0
votes
1answer
11 views

Combining different features as input to Neural Network

I use two different sources of information as input to my neural model. The model takes a word as input and produces a 1/0 output. I represent each word by using its word embedding (1024 dimensional ...
1
vote
0answers
25 views

Skip-thought models applied to phrases instead of sentences

My goal is to build a statistical model with domain specific phrase embeddings. To do this, I am doing research on how to build a model using skip-thought vectors, where instead of using sentence ...
0
votes
0answers
11 views

Doc2Vec Multiple Label Vectors

I have been exploring gensim's Doc2Vec library and it produces some pretty interesting results, and I'm beginning to explore multi-label embeddings. Through Radim's tutorial I understood that the ...
0
votes
0answers
10 views

Sequence tokenization and pretrained embedding layers

Sequence tokenization and pretrained embedding initialization - say you have a unique (but not huge) corpus of texts, and you also load a pretrained embedding vector (for example GloVe-100d). What's ...
0
votes
0answers
15 views

How to dual encode two sentences to show similarity score

I've been trying to grasp the concept of Google's semantic experiences. By using it, I'm planning to implement a semantic query tool. With universal sentence encoder I can first pre-encode all ...
0
votes
1answer
23 views

Extracting embeddings of Categorical Variables

I am trying to build a regression model, for which I have a nominal variable with very high cardinality. I am trying to get the categorical embedding of the column. Input: ...
0
votes
1answer
14 views

Tagging documents for doc2vec

I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim. What I have managed for now is to divide every text into sentence, put ...
1
vote
0answers
21 views

What is the neural network architecture behind Facebook's Starspace model?

Recently, Facebook released a paper concerning a general purpose neural embedding model called StarSpace. In their paper, they explain the loss function and the training procedure of the model, but ...
0
votes
0answers
17 views

Why does Position Embeddings work?

In the papers "Convolutional Sequence to Sequence Learning" and "Attention Is All You Need", positions embeddings are simply added to the input words embeddings to give the model a sense of the order ...
1
vote
0answers
22 views

How to learn word embedding from a context on the fly?

Consider the fictional word tahiliuk in the sentence “We found a small, fluffy tahiliuk running around our garden.” While hearing a new word used in context, people are remarkably adept at inferring a ...
0
votes
1answer
20 views

Best practice for short sentences in a deep learning network

In a deep learning network (CNN or RNN), we might use word embeddings such as FastText, Glove, etc. to represent the input text. My question is: If I'm working on a data from Twitter, and I have a ...
0
votes
0answers
23 views

How do I gather negative samples for CBOW in word2vec?

I am trying to write the cbow part of wor2vec implementation, and I am not quite sure what would be qualify as a an appropriate negative sample needed for training. Lets say we have ...
0
votes
1answer
54 views

Alternatives to doc2vec?

What are some alternatives to the doc2vec embedding model? I.e models that convert paragraphs/documents into vectors, not just models that take the mean/sum of the word embeddings of each word in the ...
0
votes
0answers
49 views

Language Models vs Word Embeddings

I wanted to clear up my understanding of both Language Models and Word Embeddings and how they are related if at all. Language Models: There is the older counting model which consists of Markov ...
1
vote
1answer
39 views

Help with reusing glove word embedding pretrained model

When using pretrained GloVe.6B for embedding generation, How can I get only the top most frequently used 100000 words rather than all the 4M words in the file?
2
votes
1answer
39 views

Predicting topics for customer reviews based on topics mapped to n-grams?

I have a large number of unlabelled customer review data(text column) and my objective is to classify each review to a particular topic. Also I have a list of unigrams,bigrams and trigrams(not a part ...
1
vote
1answer
68 views
0
votes
0answers
13 views

Long repetitive output after changing vocabulary in seq2seq model

I trained a neural question generation model, which produces sensible questions for the vocabulary that they distributed with the paper. I wanted to run the model on a different set of word embeddings ...
0
votes
1answer
91 views

doc2vec - How does the inference step work in PV-DBOW

I am quite confused about how we generate new paragraph vectors in PV-DBOW? If I want to use the embeddings to classify some text how would I generate a vector for a new paragraph? In the original ...
1
vote
3answers
45 views

Word vectors to Sentence Vectors

How can I use the vectors of words in a sentence to get the vector of that sentence . I have used strategies like - Averaging the individual word vectors or a tf-idf weighted combination of the words ....
1
vote
1answer
22 views

What are the main distribution semantics based algorithms?

I am aware that LSI, RRI and word embeddings are distributional semantics models. However, I am not certain if the below mentioned are also distributional semantic models. Non-Negative Tensor ...
0
votes
1answer
107 views

Is skip-gram model in word2vec an expanded version of N-Gram model? skip-gram vs. skip-grams?

The skip-gram model of word2vec uses a shallow neural network to learn the word embedding with (input-word, context-word) data. When I read the tutorials for the skip-gram model there was not any ...
1
vote
1answer
76 views

Ways to Encode context for text classification?

I have a binary classification task which has the following specification: Input: Chunk of text (not more than a few sentences, mostly a sentence). Additional Input: For each input sample there is ...
1
vote
1answer
31 views

Resource and useful tips on Transfer Learning in NLP

I have a few label data for training and testing a DNN. Main purpose of my work is to train a model which can do a binary classification of text. And for this purpose, I have around 3000 label data ...
1
vote
1answer
21 views

Predicting Composition of Chemical Compounds

I have a dataset which has names of compounds and their compositions. Like below Sulphuric Acid=>[H,S,O] (Hydrogen, sulphur, oxygen) Oxalic Acid=>[H,C,O] Sodium Oxalate=>[Na,C,O] Potassium Sulphate=>[...
1
vote
1answer
160 views

How to count number of word embeddings in Gensim Word2Vec model

I am trying to create a Word2Vec model of the the Pub Med Central corpus using the Gensim library and want to limit the total number of word embeddings to around 1 billion. I have searched high and ...
2
votes
0answers
16 views

Handling large word embedding matrix in Python

I have a pre trained Glove word embedding matrix (U) of dimension (400000 * 50). Now for the purpose of query expansion I need to perform the operation matmul(U*U.T). This is the term by term ...
1
vote
0answers
28 views

Learning word embeddings using RNN

The common way of learning word embeddings is based on BOW, and Skip-gram models. Is it possible to train a RNN-based architecture like GRU or LSTM with random sentences from a large corpus to learn ...
2
votes
1answer
18 views

What is max aggregation on a set of word embeddings?

In a paper I see: $\mathcal{Q}$ is a set of words. $\psi_{G^w}$ are word embeddings. so, $\{\psi_{G^w}(w_t), \forall w_t \in \mathcal{Q}\}$ gives me a set of embeddings for all words in $\mathcal{Q}...
0
votes
0answers
21 views

Unsupervised answering for a predefined set of questions

I am working on a project to read up a text segment and find answers to a specific set of questions, in order to do some information extraction. I have a set of text corpus (each of about 3000 words),...
1
vote
1answer
37 views

Can embeddings generated by word2vec be similar for words which never share any words in the same sentence?

Is it possible for word2vec to produce similar embedding vectors for two words which never share any common words in the sentences that the words are found in? Specifically, imagine I have the words ...
0
votes
1answer
66 views

Why would you use word embeddings to find similar words?

One of the applications of word embeddings (such as GloVe) is finding words of similar meaning. I just had a look at some embeddings produced by glove on large datasets and I found that the nearest ...
1
vote
2answers
323 views

How to retrain Glove Vectors on top of my own data?

I am using GloVe and gensim for my project. I have a corpus of data (let's say mydata.txt) which has new words which are not in the existing Glove. So, how do I retrain glove so that the existing pre-...
0
votes
1answer
56 views

what is the reason behind the bad outputs gained by RNN, LSTM when using GloVe pretrained model in text classification?

the problem is with the results gained for accuracy and f1 afer training our model via pretrained models such as GloVe. when I apply CNN as a classifier, the result are good as follows: ...
1
vote
0answers
124 views

Training of word weights in Word Embedding and Word2Vec

I want to know how are the word weights updated for the embedding layer in Keras and for Word2Vec. Like for the normal model.add(Embedding(..)) and ...
0
votes
0answers
136 views

is Glove better for word similarity Skip-gram/CBOW?

While looking at the slides for lecture 2 of CS224d: Deep Learning for Natural Language Processing: Link to slides It is said in slide number 31, that count based methods (ex: LSA) for creating word ...
1
vote
1answer
310 views

how to input the data set in to a word2vec by keras?

I am new in using word2vec model, as a result, I do not know how I can prepare my dataset as an input for word2vec? I have searched a lot but the datasets in tutorials were in CSV format or just one ...
0
votes
0answers
373 views

Enable Mini-batch Processing on PyTorch Word Embeddings

I am new to PyTorch and trying to create word embeddings. I started with the example below and everything works fine and it completes relatively quickly. ...
1
vote
1answer
20 views

Backpropgating error to emedding matrix

I understand the backpropagation algorithm of neural networks, and how the error propagates backwards in layers. That is, I understand that given a 3-layer feed forward network, the amount to change ...
0
votes
0answers
94 views

Case-Sensitive Word Embeddings for French

Are there any pre-trained case-sensitive word embeddings for French? The only word embeddings for French I have found is FastText and it is not case sensitive. I am currently working on problems ...
1
vote
0answers
131 views

transfer learning with sentiment analysis?

The question is how good and what are some things to keep in mind when sentiment analysis models are tested on different datasets than they are trained on. Say the task is to perform sentiment ...
1
vote
1answer
45 views

Optimal Dimension of Graph(Vertex) Embedding [closed]

Let's define a embedding of a graph structure G = (V,E) where $\mid V\mid=v, \mid E \mid=e$ Now define an embedding $f: V \to R^d$ where $d\in \Bbb N$, an optimal dimension of embedding which ...
2
votes
2answers
317 views

NN embedding layer

Several neural network libraries such as tensorflow and pytorch offer an Embedding layer. Having implemented word2vec in the past, I understand the reasoning behind wanting a lower dimensional ...
1
vote
0answers
15 views

Are there any paper for a closed domain conversational agent

i was trying to find a closed domain conversational agent/chatbot paper in Question and answering so not long conversation, and i don't think i see any. All the paper i can find are related to an ...
2
votes
2answers
474 views

Initial embeddings for unknown, padding?

Last time I've been passing pretrained word embeddings into LSTM to solve text classification problems. Usually, there are additional <pad>, ...
1
vote
0answers
24 views

How is determined the context's dimension in Doc2Vec?

I would like to know how is determined the dimension of the context in Gensim Doc2Vec.