Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

0
votes
0answers
8 views

NLP Model to predict HTML missing attributes based on Word Embedding

I am trying to build a model "doc2vec" which predicts the missed attributes (id, name) of the tag in HTML depending on the text or code around. Is that possible in doc2vec? If so, how to do it ...
1
vote
0answers
14 views

using word embedding features with linear prediction models

I have been seeing that word embedding features (e.g. here or there) are used on classification or regression tasks where the classifier/regressor is a linear one: e.g. Linear/Logistic Regressor or ...
1
vote
0answers
14 views

How to create pretrained word embedding text file with additional word features

I've had an idea for using word features to improve the quality of neural machine translation. Now, I would like to create word embeddings with additional word features such as pos tag, named entity, ...
0
votes
0answers
19 views

Reading a visualization of word embeddings

For my Masters Thesis, I created a Word2Vec model. I wanted to show this image to clarify the result. But how does the mapping works to display the words in this 2D space? All words are represented ...
0
votes
0answers
27 views

How to get the right keras model.predict label (NLP problem)

I implemented a small LSTM neural network to predict the notes for a movie. But I have an interpretation problem to convert the prob_result that ...
1
vote
0answers
16 views

Personalised search ranking for hotels

I've built hotel embeddings which gives very satisfactory results in returning similar hotels for each hotel. Now the problem I'm trying to solve is to rank the hotels in order of relevancy to the ...
0
votes
0answers
14 views

How to use document and sentence embeddings in Keras?

Although there is an easy way to use Embedding layer in keras and make use of pretrained word embeddings, is there a way to use document or sentence embeddings?
0
votes
0answers
24 views

How can I find colours in a sentence?

Given a sentence "I like blue jeans", the output should be "blue". I do not have any training data. I'll just be downloading a bunch of tweets related to a hashtag. How do I build a model for this? ...
0
votes
0answers
17 views

Problems with class embedding in keras

I am doing a toy example with mushroom dataset to learn class embedding with keras: I am trying to embed a single feature: ...
0
votes
1answer
49 views

Generating Similar Words (or Synonyms) with Word Embeddings (Word2Vec)

We have a search engine, and when users type in Tacos, we also want to search for similar words, such as Chilis or Burritos. However, it is also possible that the user search with multiple keywords. ...
0
votes
0answers
15 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
2
votes
1answer
81 views

Why does all of NLP literature use Noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples. This is difference than NCE Loss, which doesn't use a softmax at all, it uses a ...
1
vote
0answers
24 views

When should embeddings not be used for categorical data? What are their limitations?

I recently came across the concept of embeddings so the concept is still new to me, but it is my understanding that embeddings convert one-hot encoded input data into a dense vector. Vectors ...
2
votes
0answers
233 views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
1
vote
0answers
86 views

Zero padding for LSTM input

I am building a text-generation model. In the first layer, I am using Word2Vec embeddings. Now since the input is sentences they are variable length and I am padding them with zero. The input is ...
1
vote
0answers
34 views

Linear regression load model doesn't predict as expected

I have trained a linear regression model, with sklearn, for a 5 star rating and it's good enough. I have used Doc2vec to create my vectors, and saved that model. Then I save the linear regression ...
2
votes
2answers
35 views

ways to represent document by its keyword vectors

I have documents, say for example, D1, D2, D3... Dm. Every Di has its individual components or keywords k1, k2, k3,... kn, where ki is an n-dimensional vector. The number of individual components ...
0
votes
0answers
23 views

Why do we share parameters between two different inputs in the embeddings layer?

I noticed in some deep learning networks that have two inputs to the network, they use one embeddings layer to share the parameters between these two different inputs. As an example, in Keras: ...
1
vote
1answer
64 views

Dealing with multiple distinct-value categorical variables

So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values. For instance, one column have more ...
2
votes
2answers
58 views

Doc2vec for text classification task

Can I use doc2vec for classification big documents (500-2000 words, 20000 total documents, classication for three classes)? Is it a problem that the documents are large enough and contain many common ...
1
vote
1answer
135 views

Improve results using user input

I've developed a tool that retrieve the closest expressions from a database based on what the user typed. (using word embedding - a comparison is made between each expression from the database and the ...
1
vote
2answers
47 views

Doc2vec '-' symbol occurrence

Currently working on resume parser and struggled with embedding words with '-' symbols in them. Such as 'IT-manager'. Vector representations of these words are incorrectly classified by doc2vec. ['...
0
votes
1answer
81 views

What to use pretrained models (Glove) or train my own model?

I have been using pre-trained models such as google news or Glove 6B model but many words in my text data does not have their vectors representation in those pre trained model. So I was thinking of ...
1
vote
1answer
99 views

What are the approaches to aggregate categorical variables?

I am working on a clickstream dataset. I have come up with the following example dataset to explain my problem: ...
1
vote
0answers
41 views

Create word embeddings without keeping fastText Vector file in the repository

I am trying to embed a sentence with the help of Infersent, and Infersent uses fastText vectors for word embedding. The fastText vector file is close to 5 GiB. When we keep the fastText vector file ...
0
votes
1answer
255 views

What is the vector value of [CLS] [SEP] tokens in BERT

In BERT, They replace separator and start of sentence with special token labels. What are there corresponding values in embedding_matrix. Are they 0-vector? I wanted to replace the proper nouns like ...
2
votes
0answers
89 views

Keras value error: Operands could not be broadcast with with shapes(100,100) - GRU

I am trying to use Hierarchical Attention Networks for classification of news articles using 20 newsgroup dataset that i downloaded from the internet. I came across this code of the implementation and ...
5
votes
3answers
129 views

How does Doc2Vec treat numerical data which is a part of text data?

I have data containing both numbers and raw text related differently like: The power of diesel generator is 15kva. I need a single phase generator. Three phase generator required of 140 kva. Need 70g/...
1
vote
0answers
66 views

Glove supported languages

Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. I started experimenting with words embeddings also, and I found some interesting ...
0
votes
1answer
26 views

How is this tensorflow command interpreted?

I am reading an introductory tutorial on tensorflow here, and I'm confused about the code that defines an input layer for a word2vec embedding: ...
2
votes
1answer
16 views

Is there a way to cluster words based on how similarly they sound?

I have a list of words for a fictional world I've made (don't judge lol). My ultimate goal is to generate more words that sound like them through a markov generator, but for now, I'm trying to build ...
4
votes
2answers
256 views

Understanding output of LSTM for regression

Please see the update, below. I am working with embeddings and wanted to see how feasible it is to predict some scores attached to some sequences of words. The details of the scores are not important....
1
vote
0answers
14 views

Why does mean vector works better than sum or product vector?

In computing the embedding vector of a sentence or a document, I learnt that the mean of words vector is suggested, instead of sum or product. Is there a particular reason for that?
2
votes
1answer
27 views

How to learn irrelevant words in an information retrieval system?

Right now my recommender system for information retrieval uses word embedding stogether with Tfidfs weights like written here: http://nadbordrozd.github.io/blog/2016/05/20/text-classification-with-...
0
votes
0answers
15 views

define sentences with messy text data

I am extracting text from various file formats: pdf, emails, word docs, text files etc. The raw data will be processed (e.g. stemmed) but it is very likely that there are no clear sentences (e.g. ...
3
votes
1answer
53 views

How to train neural word embeddings?

So I am new to Deep Learning and NLP. I have read several blog posts on medium, towardsdatascience and papers where they talk about pre-training the word embeddings in an unsupervised fashion and then ...
0
votes
0answers
11 views

Verify compliance with the laws in text documents

USE CASE - AS IS A user sends a text document X to a system where it requires a s service (for example, to request a residence permit). A technician examines the document verifying that the document ...
1
vote
1answer
76 views

Word embeddings and punctuation symbols

I have a decent understanding of word embeddings (at its core, one can think of a word being converted into a vector of, say, 100 dimensions, and each dimension given a particular value... this allows ...
0
votes
1answer
62 views

Pretrained word embeddings vs model weights

I'm trying to understand the relationship between pretrained word vectors and pretrained weights when using pretrained word vectors in another neural network. Are the vectors themselves the weights ...
0
votes
1answer
29 views

Why does averaging a sentence's worth of word vectors work?

I am working on a text classification problem using r8-train-all-terms.txt, r8-test-all-terms.txt from https://www.cs.umb.edu/~smimarog/textmining/datasets/. The goal is to predict the label using a ...
1
vote
0answers
67 views

How to train millions of doc2vec embeddings using GPU?

I am trying to train a doc2vec based on user browsing history (urls tagged to user_id). I use chainer deep learning framework. There are more than 20 millions (user_id and urls) of embeddings to ...
1
vote
2answers
99 views

Possible reasons for word2vec learning context words as most similar rather than words in similar contexts

I am observing my word2vec model learning context words as most similar rather than words in similar contexts. I don't understand why it (word2vec in general, not my model in particular) can behave ...
1
vote
1answer
114 views

How can I parallelize GloVe reverse lookups in PyTorch?

I feel like I'm missing something obvious here because I can't find any discussion of this. I want to do a lot of reverse lookups (nearest neighbor distance searches) on the GloVe embeddings for a ...
1
vote
1answer
288 views

Updating Google News Word2vec Word Embedding?

Is it possible to update the Google News Word Embedding with a custom text dataset (text data pertaining to a particular domain) ? Google News Word2Vec - Word Embedding clearly helps us to come with ...
3
votes
1answer
168 views

Why ELMo's word embedding can represent the word better than glove?

I have read the code of ELMo: https://github.com/allenai/bilm-tf Based on my understanding, ELMo first init an word embedding matrix A for all the word and then ...
0
votes
1answer
66 views

Combining different features as input to Neural Network

I use two different sources of information as input to my neural model. The model takes a word as input and produces a 1/0 output. I represent each word by using its word embedding (1024 dimensional ...
1
vote
0answers
48 views

Skip-thought models applied to phrases instead of sentences

My goal is to build a statistical model with domain specific phrase embeddings. To do this, I am doing research on how to build a model using skip-thought vectors, where instead of using sentence ...
1
vote
1answer
109 views

How to dual encode two sentences to show similarity score

I've been trying to grasp the concept of Google's semantic experiences. By using it, I'm planning to implement a semantic query tool. With universal sentence encoder I can first pre-encode all ...
1
vote
1answer
374 views

Extracting embeddings of Categorical Variables

I am trying to build a regression model, for which I have a nominal variable with very high cardinality. I am trying to get the categorical embedding of the column. Input: ...
0
votes
1answer
399 views

Tagging documents for doc2vec

I am working on resume parsing script. I am trying to tag documents sentences with TaggedDocument function, provided by gensim. What I have managed for now is to divide every text into sentence, put ...