Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

How does embedding layer works in pytorch with neural machine translation?

as i mentioned on title, How does pytorch embedding layer works in machine translation task ? As i know that we can use CBOW or Skip-gram to create pretrained embedding vectors for our translation ...
0
votes
0answers
22 views

Mathematic vs Neural Network Approach for creating word vectors for a corpus of text

Are there particular advantages or disadvantages for using word2vec(neural nets) rather than Pointwise Mutual Information(PMI) and Singular Value Decomposition(SVD)(mathematical approach) for the ...
1
vote
1answer
25 views

NLP - Identify Tagged Words

Please pardon me as the title might not be very accurate I am trying to create a model that learns the word representation and then is able to predict word representation in another piece of text. An ...
0
votes
0answers
14 views

Doc2Vec for multi class classification

I am working on my first project, I am trying to predict the quality of a software specification requirements. I have 1000 requirements which have been manually labelled on a scale of 1-5 (poor-...
0
votes
0answers
19 views

BERT or ELMo for Document Similarity

Does anyone use BERT or ELMo language models to determine the similarity between two text documents? My question aims to collect all possible ways for combining the contextual word embeddings ...
2
votes
0answers
30 views

Predicting word from a set of words

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
1
vote
2answers
21 views

Word embedding of a new word which was not in training

Let's say I trained a Skip-Gram model (Word2Vec) for my vocabulary of size 10,000. The representation allows me to reduce the dimension from 10,000 (one-hot-encoding) to 100 (size of hidden layer of ...
0
votes
0answers
10 views

threshold for word/embeddings based on frequency in DNNLinearCombinedClassifier

I'm using Tensorflow's DNNLinearCombinedClassifier for multi-class classification. Irrespective of my vocabulary size I'm ...
0
votes
0answers
5 views

Usage doubt of tf.nn.nce_loss!

tf.nn.nce_loss, beautifully explained here Understanding tf.nn.nce_loss() in tensorflow, but still this method always confuse me when I compare with its actual ...
0
votes
1answer
10 views

Do word embeddings help with out of vocab tokens?

I am performing sentiment analysis on a custom dataset of text with Keras but am a little confused about word embeddings. I have been able to train an "Embedding" layer and have also learned to load ...
0
votes
1answer
24 views

How to utilize dictionary data set for text classification?

I have a dataset similar to newsgroup20 for classification. With the training dataset, I have a dictionary data set that explains some jargons in the training dataset. These both are different data ...
0
votes
0answers
15 views

Is Elmo equivalent to Fasttext+Bi-directional GRU?

From what I have read, Elmo uses bi-directional LSTM layers to give contextual embeddings for words in a sentence. So if I use a bi-directional LSTM/GRU layer over Fasttext representations of words, ...
4
votes
1answer
69 views

Reason: Average word vector embedding encodes word content and word order effectively

I was going through a research paper: FINE-GRAINED ANALYSIS OF SENTENCE EMBEDDINGS USING AUXILIARY PREDICTION TASKS The key take away was Comparison of Encoder decoder and average word sentence ...
0
votes
0answers
10 views

NLP text one-hot encoding with mapping dataset

I am wondering if I can make NLP model to compute word similarity with using data consisting of mapping data. I think I can make a model learn with vectorized words by one-hot encoding. Is is possible?...
1
vote
0answers
37 views

BERT vs Word2VEC: Is bert disambuguate the meaning of the word vector?

Word2vec: Word2vec provides a vector for each token/word and those vectors encode the meaning of the word. Although those vectors are not human interpretable, the meaning of the vectors are ...
3
votes
2answers
64 views

What is the difference between and Embedding Layer and an Autoencoder?

I'm reading about Embedding layers, especially applied to NLP and word2vec, and they seem nothing more than an application of Autoencoders for dimensionality reduction. Are they different? If so, what ...
1
vote
2answers
31 views

Word Embedding for Item Names(integer, one-hot encoding)

I am looking for the way to get the similarity between two item names using integer encoding or one-hot encoding. For example, "lane connector" vs. "a truck crane". I have 100,000 item names ...
1
vote
1answer
28 views

Why use separable convolutions on one channel input?

I'm currently working on the Text Classification Guide from Google. During step 4, they create a CNN with separable convolutions for use with word embeddings: <...
1
vote
1answer
23 views

Can word embedding be used for text classification on a mix of English and non-English text?

I'm doing text classification on text messages generated by consumers and just realized even though most of the replies provided by consumers are in English, some are in French. I've used Keras word ...
0
votes
1answer
22 views

NLP: robust ways to handle morphological variations in words (e.g. plurals, verb conjugations, hyphens, etc.)?

I need to process natural language sentences in which words can appear with morphological variations: car -> cars; play -> playing, played; etc. There might be hyphens also, e.g. "dog-friendly hotel", ...
1
vote
0answers
25 views

Copying embeddings for gensim word2vec

I wanted to see if I can simply set new weights for gensim's Word2Vec without training. I get the 20 News Group data set from scikit-learn (from sklearn.datasets import fetch_20newsgroups) and trained ...
1
vote
0answers
18 views

Learning Embeddings for One Word

I have a non-conventional NLP task. I am looking to develop a sequence to a vector model. Instead of employing one-hot encoding ...
2
votes
1answer
13 views

What are the equations involved in calculation of the parameters of embedding layer?

I'm trying to perform sentiment analysis on some data using keras.I'm using embedding layer and then LSTM. I know that embedding layer decreases the sparsity of the one hot encodings of the words and ...
2
votes
1answer
21 views

Why using a frozen embedding layer in an LSTM model

I'm studying this LSTM mode: https://www.kaggle.com/paoloripamonti/twitter-sentiment-analysis They use a frozen embedding layer which uses an predefined matrix with for each word a 300 dim vector ...
0
votes
0answers
21 views

How to plot a wordCloud for essay text from a confusion matrix false positive rate count?

I have an essay of text(BOW) and I have modelled it using let's say any model and plotted the confusion matrix and that I have got FPR, I need to plot a word cloud which shows the words due to which ...
1
vote
1answer
42 views

word2vec word embeddings creates very distant vectors, closest similarity is still very far

I started using gensim's FastText to create word embeddings on a large corpus of a specialized domain (after finding that existing open source embeddings are not performing well on this domain), ...
1
vote
1answer
17 views

Keras Embedding Layer main paper

I need to explain the word embedding layer of Keras in my paper, mathematically. I know that keras initialize the embedding vectors randomly and then update the parameters using the optimizer ...
2
votes
0answers
65 views

Bert: fine-tuning the entire pre-trained model end-to-end vs using contextual token vector

In the official github page of BERT, it mentions that: In certain cases, rather than fine-tuning the entire pre-trained model end-to-end, it can be beneficial to obtained pre-trained contextual ...
2
votes
0answers
32 views

meaning of fine-tuning in nlp task

There are two types of transfer learning model. One is feature extraction, where the weights of the pre-trained model are not changed while training on the actual task and other is the weights of the ...
0
votes
0answers
9 views

NLP Model to predict HTML missing attributes based on Word Embedding

I am trying to build a model "doc2vec" which predicts the missed attributes (id, name) of the tag in HTML depending on the text or code around. Is that possible in doc2vec? If so, how to do it ...
1
vote
0answers
25 views

using word embedding features with linear prediction models

I have been seeing that word embedding features (e.g. here or there) are used on classification or regression tasks where the classifier/regressor is a linear one: e.g. Linear/Logistic Regressor or ...
1
vote
0answers
17 views

How to create pretrained word embedding text file with additional word features

I've had an idea for using word features to improve the quality of neural machine translation. Now, I would like to create word embeddings with additional word features such as pos tag, named entity, ...
0
votes
0answers
21 views

Reading a visualization of word embeddings

For my Masters Thesis, I created a Word2Vec model. I wanted to show this image to clarify the result. But how does the mapping works to display the words in this 2D space? All words are represented ...
0
votes
0answers
36 views

How to get the right keras model.predict label (NLP problem)

I implemented a small LSTM neural network to predict the notes for a movie. But I have an interpretation problem to convert the prob_result that ...
1
vote
0answers
17 views

Personalised search ranking for hotels

I've built hotel embeddings which gives very satisfactory results in returning similar hotels for each hotel. Now the problem I'm trying to solve is to rank the hotels in order of relevancy to the ...
0
votes
0answers
19 views

How to use document and sentence embeddings in Keras?

Although there is an easy way to use Embedding layer in keras and make use of pretrained word embeddings, is there a way to use document or sentence embeddings?
0
votes
0answers
25 views

How can I find colours in a sentence?

Given a sentence "I like blue jeans", the output should be "blue". I do not have any training data. I'll just be downloading a bunch of tweets related to a hashtag. How do I build a model for this? ...
0
votes
0answers
31 views

Problems with class embedding in keras

I am doing a toy example with mushroom dataset to learn class embedding with keras: I am trying to embed a single feature: ...
0
votes
1answer
122 views

Generating Similar Words (or Synonyms) with Word Embeddings (Word2Vec)

We have a search engine, and when users type in Tacos, we also want to search for similar words, such as Chilis or Burritos. However, it is also possible that the user search with multiple keywords. ...
0
votes
0answers
20 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
2
votes
1answer
130 views

Why does all of NLP literature use Noise contrastive estimation loss for negative sampling instead of sampled softmax loss?

A sampled softmax function is like a regular softmax but randomly selects a given number of 'negative' samples. This is difference than NCE Loss, which doesn't use a softmax at all, it uses a ...
1
vote
0answers
35 views

When should embeddings not be used for categorical data? What are their limitations?

I recently came across the concept of embeddings so the concept is still new to me, but it is my understanding that embeddings convert one-hot encoded input data into a dense vector. Vectors ...
3
votes
0answers
877 views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
1
vote
0answers
166 views

Zero padding for LSTM input

I am building a text-generation model. In the first layer, I am using Word2Vec embeddings. Now since the input is sentences they are variable length and I am padding them with zero. The input is ...
1
vote
0answers
36 views

Linear regression load model doesn't predict as expected

I have trained a linear regression model, with sklearn, for a 5 star rating and it's good enough. I have used Doc2vec to create my vectors, and saved that model. Then I save the linear regression ...
2
votes
2answers
39 views

ways to represent document by its keyword vectors

I have documents, say for example, D1, D2, D3... Dm. Every Di has its individual components or keywords k1, k2, k3,... kn, where ki is an n-dimensional vector. The number of individual components ...
0
votes
0answers
24 views

Why do we share parameters between two different inputs in the embeddings layer?

I noticed in some deep learning networks that have two inputs to the network, they use one embeddings layer to share the parameters between these two different inputs. As an example, in Keras: ...
1
vote
1answer
155 views

Dealing with multiple distinct-value categorical variables

So, I've got a dataset with almost all of its columns are categorical variables. Problem is that most of the categorical variables have so many distinct values. For instance, one column have more ...
2
votes
2answers
73 views

Doc2vec for text classification task

Can I use doc2vec for classification big documents (500-2000 words, 20000 total documents, classication for three classes)? Is it a problem that the documents are large enough and contain many common ...
1
vote
1answer
190 views

Improve results using user input

I've developed a tool that retrieve the closest expressions from a database based on what the user typed. (using word embedding - a comparison is made between each expression from the database and the ...