Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

1
vote
0answers
9 views

Keras value error: Operands could not be broadcast with with shapes(100,100) - GRU

I am trying to use Hierarchical Attention Networks for classification of news articles using 20 newsgroup dataset that i downloaded from the internet. I came across this code of the implementation and ...
3
votes
1answer
32 views

How does Doc2Vec treat numerical data which is a part of text data?

I have data containing both numbers and raw text related to different like: The power of diesel generator is 15kva. I need a single phase generator. Three phase generator required of 140 kva. Need ...
1
vote
0answers
24 views

Glove supported languages

Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. I started experimenting with words embeddings also, and I found some interesting ...
0
votes
1answer
23 views

How is this tensorflow command interpreted?

I am reading an introductory tutorial on tensorflow here, and I'm confused about the code that defines an input layer for a word2vec embedding: ...
2
votes
1answer
15 views

Is there a way to cluster words based on how similarly they sound?

I have a list of words for a fictional world I've made (don't judge lol). My ultimate goal is to generate more words that sound like them through a markov generator, but for now, I'm trying to build ...
4
votes
2answers
156 views
+50

Understanding output of LSTM for regression

Please see the update, below. I am working with embeddings and wanted to see how feasible it is to predict some scores attached to some sequences of words. The details of the scores are not important....
1
vote
0answers
11 views

Why does mean vector works better than sum or product vector?

In computing the embedding vector of a sentence or a document, I learnt that the mean of words vector is suggested, instead of sum or product. Is there a particular reason for that?
2
votes
1answer
18 views

How to learn irrelevant words in an information retrieval system?

Right now my recommender system for information retrieval uses word embedding stogether with Tfidfs weights like written here: http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-...
0
votes
0answers
12 views

define sentences with messy text data

I am extracting text from various file formats: pdf, emails, word docs, text files etc. The raw data will be processed (e.g. stemmed) but it is very likely that there are no clear sentences (e.g. ...
3
votes
1answer
44 views

How to train neural word embeddings?

So I am new to Deep Learning and NLP. I have read several blog posts on medium, towardsdatascience and papers where they talk about pre-training the word embeddings in an unsupervised fashion and then ...
0
votes
0answers
11 views

Verify compliance with the laws in text documents

USE CASE - AS IS A user sends a text document X to a system where it requires a s service (for example, to request a residence permit). A technician examines the document verifying that the document ...
1
vote
1answer
27 views

Word embeddings and punctuation symbols

I have a decent understanding of word embeddings (at its core, one can think of a word being converted into a vector of, say, 100 dimensions, and each dimension given a particular value... this allows ...
0
votes
1answer
27 views

Pretrained word embeddings vs model weights

I'm trying to understand the relationship between pretrained word vectors and pretrained weights when using pretrained word vectors in another neural network. Are the vectors themselves the weights ...
0
votes
1answer
24 views

Why does averaging a sentence's worth of word vectors work?

I am working on a text classification problem using r8-train-all-terms.txt, r8-test-all-terms.txt from https://www.cs.umb.edu/~smimarog/textmining/datasets/. The goal is to predict the label using a ...
0
votes
0answers
20 views

Embedding when using an RNN and zero-padding on input strings

I have started developing an RNN/LSTM in tensorflow to take in short sentences (typically of length 5-15 tokens) along with a second categorical variable. The goal is to create an encoder-decoder to ...
1
vote
0answers
28 views

How to train millions of doc2vec embeddings using GPU?

I am trying to train a doc2vec based on user browsing history (urls tagged to user_id). I use chainer deep learning framework. There are more than 20 millions (user_id and urls) of embeddings to ...
0
votes
0answers
13 views

Word embedding based on a tree structure

I need to train a model for MeSH Terms. The similarity metric I want to use is MeSH Tree Structure. Every MeSH term has one or more tree number(s), e.g.: Term, Tree Number(s): ...
1
vote
2answers
59 views

Possible reasons for word2vec learning context words as most similar rather than words in similar contexts

I am observing my word2vec model learning context words as most similar rather than words in similar contexts. I don't understand why it (word2vec in general, not my model in particular) can behave ...
1
vote
1answer
35 views

How can I parallelize GloVe reverse lookups in PyTorch?

I feel like I'm missing something obvious here because I can't find any discussion of this. I want to do a lot of reverse lookups (nearest neighbor distance searches) on the GloVe embeddings for a ...
1
vote
1answer
109 views

Updating Google News Word2vec Word Embedding?

Is it possible to update the Google News Word Embedding with a custom text dataset (text data pertaining to a particular domain) ? Google News Word2Vec - Word Embedding clearly helps us to come with ...
3
votes
1answer
98 views

Why ELMo's word embedding can represent the word better than glove?

I have read the code of ELMo: https://github.com/allenai/bilm-tf Based on my understanding, ELMo first init an word embedding matrix A for all the word and then ...
0
votes
1answer
32 views

Combining different features as input to Neural Network

I use two different sources of information as input to my neural model. The model takes a word as input and produces a 1/0 output. I represent each word by using its word embedding (1024 dimensional ...
1
vote
0answers
32 views

Skip-thought models applied to phrases instead of sentences

My goal is to build a statistical model with domain specific phrase embeddings. To do this, I am doing research on how to build a model using skip-thought vectors, where instead of using sentence ...
1
vote
1answer
59 views

How to dual encode two sentences to show similarity score

I've been trying to grasp the concept of Google's semantic experiences. By using it, I'm planning to implement a semantic query tool. With universal sentence encoder I can first pre-encode all ...
1
vote
1answer
168 views

Extracting embeddings of Categorical Variables

I am trying to build a regression model, for which I have a nominal variable with very high cardinality. I am trying to get the categorical embedding of the column. Input: ...
0
votes
1answer
160 views

Tagging documents for doc2vec

I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim. What I have managed for now is to divide every text into sentence, put ...
1
vote
0answers
41 views

What is the neural network architecture behind Facebook's Starspace model?

Recently, Facebook released a paper concerning a general purpose neural embedding model called StarSpace. In their paper, they explain the loss function and the training procedure of the model, but ...
1
vote
0answers
128 views

Why does Position Embeddings work?

In the papers "Convolutional Sequence to Sequence Learning" and "Attention Is All You Need", positions embeddings are simply added to the input words embeddings to give the model a sense of the order ...
1
vote
0answers
24 views

How to learn word embedding from a context on the fly?

Consider the fictional word tahiliuk in the sentence “We found a small, fluffy tahiliuk running around our garden.” While hearing a new word used in context, people are remarkably adept at inferring a ...
0
votes
1answer
23 views

Best practice for short sentences in a deep learning network

In a deep learning network (CNN or RNN), we might use word embeddings such as FastText, Glove, etc. to represent the input text. My question is: If I'm working on a data from Twitter, and I have a ...
0
votes
1answer
113 views

Alternatives to doc2vec?

What are some alternatives to the doc2vec embedding model? I.e models that convert paragraphs/documents into vectors, not just models that take the mean/sum of the word embeddings of each word in the ...
1
vote
1answer
82 views

Help with reusing glove word embedding pretrained model

When using pretrained GloVe.6B for embedding generation, How can I get only the top most frequently used 100000 words rather than all the 4M words in the file?
2
votes
1answer
54 views

Predicting topics for customer reviews based on topics mapped to n-grams?

I have a large number of unlabelled customer review data(text column) and my objective is to classify each review to a particular topic. Also I have a list of unigrams,bigrams and trigrams(not a part ...
1
vote
1answer
115 views
0
votes
0answers
19 views

Long repetitive output after changing vocabulary in seq2seq model

I trained a neural question generation model, which produces sensible questions for the vocabulary that they distributed with the paper. I wanted to run the model on a different set of word embeddings ...
0
votes
1answer
352 views

doc2vec - How does the inference step work in PV-DBOW

I am quite confused about how we generate new paragraph vectors in PV-DBOW? If I want to use the embeddings to classify some text how would I generate a vector for a new paragraph? In the original ...
2
votes
3answers
119 views

Word vectors to Sentence Vectors

How can I use the vectors of words in a sentence to get the vector of that sentence . I have used strategies like - Averaging the individual word vectors or a tf-idf weighted combination of the words ....
1
vote
1answer
23 views

What are the main distribution semantics based algorithms?

I am aware that LSI, RRI and word embeddings are distributional semantics models. However, I am not certain if the below mentioned are also distributional semantic models. Non-Negative Tensor ...
0
votes
1answer
157 views

Is skip-gram model in word2vec an expanded version of N-Gram model? skip-gram vs. skip-grams?

The skip-gram model of word2vec uses a shallow neural network to learn the word embedding with (input-word, context-word) data. When I read the tutorials for the skip-gram model there was not any ...
2
votes
1answer
163 views

Ways to Encode context for text classification?

I have a binary classification task which has the following specification: Input: Chunk of text (not more than a few sentences, mostly a sentence). Additional Input: For each input sample there is ...
1
vote
1answer
42 views

Resource and useful tips on Transfer Learning in NLP

I have a few label data for training and testing a DNN. Main purpose of my work is to train a model which can do a binary classification of text. And for this purpose, I have around 3000 label data ...
1
vote
1answer
25 views

Predicting Composition of Chemical Compounds

I have a dataset which has names of compounds and their compositions. Like below Sulphuric Acid=>[H,S,O] (Hydrogen, sulphur, oxygen) Oxalic Acid=>[H,C,O] Sodium Oxalate=>[Na,C,O] Potassium Sulphate=>[...
1
vote
1answer
257 views

How to count number of word embeddings in Gensim Word2Vec model

I am trying to create a Word2Vec model of the the Pub Med Central corpus using the Gensim library and want to limit the total number of word embeddings to around 1 billion. I have searched high and ...
2
votes
0answers
18 views

Handling large word embedding matrix in Python

I have a pre trained Glove word embedding matrix (U) of dimension (400000 * 50). Now for the purpose of query expansion I need to perform the operation matmul(U*U.T). This is the term by term ...
1
vote
0answers
61 views

Learning word embeddings using RNN

The common way of learning word embeddings is based on BOW, and Skip-gram models. Is it possible to train a RNN-based architecture like GRU or LSTM with random sentences from a large corpus to learn ...
2
votes
1answer
28 views

What is max aggregation on a set of word embeddings?

In a paper I see: $\mathcal{Q}$ is a set of words. $\psi_{G^w}$ are word embeddings. so, $\{\psi_{G^w}(w_t), \forall w_t \in \mathcal{Q}\}$ gives me a set of embeddings for all words in $\mathcal{Q}...
0
votes
0answers
21 views

Unsupervised answering for a predefined set of questions

I am working on a project to read up a text segment and find answers to a specific set of questions, in order to do some information extraction. I have a set of text corpus (each of about 3000 words),...
1
vote
1answer
39 views

Can embeddings generated by word2vec be similar for words which never share any words in the same sentence?

Is it possible for word2vec to produce similar embedding vectors for two words which never share any common words in the sentences that the words are found in? Specifically, imagine I have the words ...
0
votes
1answer
123 views

Why would you use word embeddings to find similar words?

One of the applications of word embeddings (such as GloVe) is finding words of similar meaning. I just had a look at some embeddings produced by glove on large datasets and I found that the nearest ...
1
vote
2answers
610 views

How to retrain Glove Vectors on top of my own data?

I am using GloVe and gensim for my project. I have a corpus of data (let's say mydata.txt) which has new words which are not in the existing Glove. So, how do I retrain glove so that the existing pre-...