Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
0
votes
1answer
25 views

Problem when using Autograd with nn.Embedding in Pytorch

I am in trouble with taking derivatives of outputs logits with respect to the inputs input_ids. Here is an example of my input: ...
0
votes
1answer
9 views

Flair Embeddings - Significance of Backwards vs Forwards?

I'm working on a project that makes use of Flair for stacked embeddings. I'm looking at the built in embeddings on this page. I noticed that the table shows news-X ...
0
votes
0answers
10 views

How effective would this pseudo-LDA2Vec implementation be?

For my site I'm working on a chat recommender that would recommend chats to users. Each chat has a title and description and my corpus is composed of many of these title and description documents. I ...
1
vote
0answers
18 views

Semi-Supervised Learning using NLP

I am working on a drug reaction problem in which I need to extract tweets and label the tweets (binary-reaction due to drug or not). But since I don't have domain knowledge, and clustering would also ...
1
vote
0answers
12 views

Embedding of the item

I've got a question about embedding. I've got dataset of items of which I have a name, description and image. I want to make vector representation of them using these three features. How should I do ...
0
votes
0answers
33 views

why does adding an LDA document vector with a word2vec word vector work well in LDA2vec?

In LDA the document weight vector represents the "weights" of each topic in the document. I think it's also valid to say, each row in the document vector corresponds to a word in the document, the ...
1
vote
0answers
20 views

Keras : Classifying names per origin using RNN and an embedding layer

I am trying to classify names with character RNN using embedding (similar to the PyTorch example https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html) The problem is that ...
0
votes
1answer
29 views

Why does accuracy improve when vocabulary size of word embedding model is reduced?

I am quite new to the topic of word embedding using word2vec and models such as skip-gram. Our teacher introduced us to this TensorFlow code on word2vec which he ran on Google Colab notebook. He ...
1
vote
0answers
16 views

Character-level embeddings in python

I'm working on an NLP task that requires the use of character level embeddings, and I've been trying to use Spacy. However, it seems that spacy uses word-level embeddings for the word vectors, and I ...
1
vote
1answer
31 views

Why does averaging over vector values cause errors?

As a way to improve my model, I want to average GloVe vectors over a sentence. However, I can't get np.mean to work. The following code works when not averaging over words. (copied from other code) <...
0
votes
0answers
11 views

How I change Pre-trained word embedding model using CBOW to Skip-gram

Is it possible to change the parameters of the model 'cc.ar.300.vec' from CBOW to Skip-gram with the dimension of 100 using Python code?
3
votes
2answers
172 views

Building a tag-based recommendation engine given a set of user tags?

Basically, the idea is to have users following tags on the site, so each users has a set of tags they are following. And then there is a document collection where each document in the collection has a ...
1
vote
1answer
40 views

Would averaging two vectors in word embeddings make sense?

I'm currently using the GloVe embedding matrix which is pre-trained on a large corpus. For my purpose it works fine, however, there are a few words which it does not know (for example, the word '...
0
votes
0answers
7 views

Calculating semantic coherence in a given speech transcript

I am trying to calculate the semantic coherence in a given paragraph/transcript, ie. if somebody goes off track while talking about a thing or topic - more specifically describing a picture (the ...
2
votes
0answers
17 views

Training pipelines where featurization/NLP is more expensive than backprop

I'm working on a document classification project and I'm using a neural net in tensorflow, where the features are 300-dimensional word embeddings, either from fastext or word2vec (yes I know that ...
2
votes
1answer
76 views

Distribution plot of word embeddings

I have a list of sentences and a 10 dimensional embedding for each of the sentence. I am trying to visualize these embeddings so that i can see if several sentence embeddings form a cluster such ...
1
vote
0answers
25 views

How to train a Word2vec model so that particular words set would reside close to each other in the vector space?

I am supposed to build a resume parser. For the skill extraction part, currently I am matching bi-grams and uni-grams in a CV against a predefined skill set, which is not that successful. Can I train ...
0
votes
0answers
16 views

Get row wise frequency count of words from list in text column pandas

I have a data frame with a Audio Transcript column from customer care phone conversation. I have created one list with words and sentences ...
-1
votes
1answer
29 views

LSTM for text with different sentences size, but same input-output sizes

Hello fellow Data Scientists I'm trying to use a LSTM (using word embeddings) to generate a system that can tag each word of a sentence. For this, I give it a set of sentences of different sizes and ...
0
votes
0answers
9 views

Add new category/word to Keras embedding layer

I have a Keras model that is regularly updated by running a few additional epochs on new data once it becomes available. Part of the model is an embedding layer for a categorical feature. Recently ...
0
votes
0answers
17 views

How to Increase Low Accuracy Keras Model Design?

I am trying to use the Embedding layer with CNN. Also, use train data 1200 and test data 300. use Keras model design. At this moment, the accuracy achieved 40% or 50%. For this reason, My Question is ...
1
vote
0answers
12 views

More pre-trained embeddings for PyTorch Big Graph

Apart from the entire wikidata, are there any other PyTorch Big Graph pre-trained graph embeddings on smaller sized knowledge graph, like freebase-15k? I do not have the resources to build it from ...
2
votes
1answer
116 views

Does Fasttext use One Hot Encoding?

In the original Skipgram/CBOW both context word and target word are represented as one-hot encoding. Does fasttext also use one-hot encoding for each subword when training the skip-gram/CBOW model (...
2
votes
1answer
44 views

How to get sentence from embedding vector with Universal Sentence Encoder?

I'd like to ask, if there is possibility to get sentence (or word) from embedding vector using Universal Sentence Encoder? First of all, I've clustered my embedded sentences and I've got a vector ...
0
votes
1answer
99 views

How to preprocess data for Word2Vec?

I have text data which is crawled from websites. I am preprocessing data to train Word2Vec model. Should I remove stopwords and do lemmatization? How to preprocess data for Word2Vec?
0
votes
0answers
19 views

Learn recommendations from dataset

I have a corpus of news articles and 6 annotations indicating whether a pair of documents are related or not. Not all possible pairs are annotated. Since a vectorised representation can find out the ...
0
votes
0answers
42 views

Semantic Search Help

There is a problem we are trying to solve where we want to do semantic search on our set of data, i.e we have a domain specific data (example: sentences talking about automobiles) Our data is just a ...
0
votes
0answers
8 views

Adding feed forward layer for word2vec/doc2vec in hidden layers

Word2vec and paragraph2vec(doc2vec) both adopt very simple strucutre -- input layer, hidden layer which concatenate or averaging the input layer, and softmax output layer. If one add more hidden ...
3
votes
2answers
93 views

What are the elements in a BERT word embedding?

As far as I understand, BERT is a word embedding that can be fine-tuned or used directly. With older word embeddings (word2vec, Glove), each word was only represented once in the embedding (one ...
0
votes
1answer
77 views

Using Trainable=True in Keras Embedding obtained better performance

It is suggested by the author of Keras [1] to use Trainable=False when using the embedding layer in Keras to prevent the weights from being updated during training. ...
1
vote
0answers
16 views

Are vectors generated by doc2vec and similar models uniformly distributed?

I have read that vectors in a word2vec model are very much not uniformly distributed and are thought to follow Zipf's law; is this the same for the associated models like paragraph2vec, doc2vec, etc? ...
3
votes
1answer
46 views

BERT word embedings for finding word definition

Can BERT, GPT or other contextualised embedings be used for finding word definitions? What would be the most effective and not complicated approach for tackling a sample task as described below. Map ...
1
vote
1answer
20 views

How do I encode time in high dimensional space?

I have a dataset of form text, text, category, category, time, text and I would like to apply the attention mechanism to it. This requires that all inputs be in the ...
2
votes
1answer
37 views

Word Embedding or Hash?

In my dataset I have a 'text' column and a 'followers' column containing lists of follower IDs, i.e. '1093777852477116417, 936194589043683328,...'. Some of the 'followers' values contain thousands of ...
2
votes
0answers
37 views

Building own embedding from sequence

I have 100 sequences of the word (i.e., action for completing a task). Each of the sequences contains around 350 actions(115 unique actions but all the actions are not used in each sequence. Some of ...
0
votes
0answers
26 views

How to create POS Embeddings for input?

I am trying to append POS embeddings to an input word embedding (which is from Glove). In order to do that, I need represent POS tags as embeddings. I saw the link in this channel already; however I ...
1
vote
3answers
38 views

Categorical variables with multiple entries transformed to entity embedding

I have structured data with lots (tens of thousads) of categories organized into columns. The goal is to enter the data into gradient boosting machine algorithm for a specific prediction. Some ...
1
vote
1answer
35 views

(pre-trained) python package for semantic word similarity

I am searching for a python package that calculates the semantic similarity between words. I do not want to train a model (what most packages seem to offer) - the package should have been pre-trained ...
0
votes
0answers
59 views

How to input sentence embedding as a feature in Neural Networks?

After preprocessing, I have the pandas dataframe like in the screenshot attached. I want to train a Keras neural network model to classify the vectors into intents. The vectors here are LASER ...
1
vote
0answers
14 views

How StarSpace works

I have gone through StarSpace algorithm but unable to understand how the output dimension is compared to Y (target variable) so that it can output the dimensions and loss function is optimized.
1
vote
1answer
21 views

How to incorporate keyboard positions on character level embeddings?

I am working with NLP and have character level embeddings. I have embeddings learned from Wikipedia text. Now, I want to learn embeddings from chat data (where misspellings and abbreviations are way ...
1
vote
1answer
22 views

How does backpropagation work with averaging layers?

I'm studying Word2Vec algorithm, and so far i understood that, in the case of input context bigger than 1 (so multiple words) we have our hidden layer that performs averaging between the inputs (as ...
2
votes
1answer
67 views

Why is T test reweighting on a word X word co-occurrence matrix so effective?

I am going through Stanford NLP class: http://web.stanford.edu/class/cs224u/ A task in the homework is to implement T-test reweighting on a word X word co-occurrence matrix: https://nbviewer.jupyter....
2
votes
1answer
55 views

Can 2 different OOV words get the same vector in FastText?

Since FastText sums up the vectors(order is not considered) of an OOV word's subwords, is it possible for two different OOV words to get the same vector ? If so, then can you give an example?
1
vote
1answer
257 views

Proper masking in the transformer model

Concerning the transformer model, a mask is used to mask out attention scores (replace with 1e-9) prior to the matrix multiplication with the value tensor. Regarding the masking, I have 3 short ...
0
votes
1answer
55 views

What's wrong with RF/SVM with word embedding (GloVe)?

I searched many times in google for examples on word embedding (specifically GloVe) with Random forest and I couldn't find any single example. For GloVe, it was all either LSTM or CNN. Maybe there's ...
5
votes
1answer
99 views

Would Topic Modelling be classified as NLP or NLU?

I recently started my journey into the world of NLP, it's been one heck of a ride. I'm currently trying to understand whether topic modelling would be considered as NLP or NLU. Initially I would ...
1
vote
0answers
15 views

Keras Embedding layer weights

I am trying to analyze the weights in a trained Keras embedding layer. The goal is to understand how well (or how poorly) we're capturing relationships between some specific tokens in our texts, ...
1
vote
3answers
42 views

group the similar words

array(['Ruby on Rails', 'Ruby', 'AWS DynamoDB', 'Python', 'MySQL', 'Swift', 'Android', 'iOS', 'JavaScript', 'React Native', 'ReactJS', 'TypeScript', 'Vue.js', 'Webpack', 'Amazon Web ...
2
votes
1answer
61 views

Difficulty interpreting word embedding vector similarity (spaCy)

I calculate vector similarities like this: ...

1
2 3 4 5 6