Skip to main content

Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

157 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
7 votes
1 answer
9k views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
Harman's user avatar
  • 706
4 votes
1 answer
357 views

Does Word2Vec's skip-gram NNLM even produce context words?

Let me first establish what CBoW and skip-gram are supposed to do. You can skip to the next section if you think this is unnecessary. Background My understanding is that Word2Vec is a suite of 2 ...
Mew's user avatar
  • 243
3 votes
0 answers
441 views

Text embeddings for words or very short sentences with a LLM

I tried to compute the semantic similarity between words or short sentences. Ex : inflation vs price raising I have tried the openai embeddings API and cosine distance but the results are very poor. I ...
user2479920's user avatar
3 votes
2 answers
836 views

Gensim doc2vec error: KeyError: "word 'senseless' not in vocabulary"

I am new to machine learning and tried doc2vec on quora duplicate dataset. new_dfx has columns 'question1' and 'question2' which has preprocessed questions in each row. Following is the tagged ...
Ankit Rohilla's user avatar
3 votes
0 answers
190 views

Medical NER for French language

I'm currently exploring the options to extract medical NER specifically for French language. I tried SpaCy's general French NER but it wasn't helpful to the cause (...
Van Peer's user avatar
  • 285
3 votes
1 answer
921 views

Character-level embeddings in python

I'm working on an NLP task that requires the use of character level embeddings, and I've been trying to use Spacy. However, it seems that spacy uses word-level embeddings for the word vectors, and I ...
rmaguiar's user avatar
  • 163
3 votes
1 answer
229 views

How to build a symmetric similarity model on top of embeddings?

I have two equal length vectors that come out of two identical embedding layers. I want to calculate their similarity, and I don't trust the embedding layer enough to just use dot product (e.g. it's ...
Uri's user avatar
  • 131
3 votes
2 answers
223 views

How to train neural word embeddings?

So I am new to Deep Learning and NLP. I have read several blog posts on medium, towardsdatascience and papers where they talk about pre-training the word embeddings in an unsupervised fashion and then ...
Vatsal Aggarwal's user avatar
3 votes
1 answer
534 views

How to train millions of doc2vec embeddings using GPU?

I am trying to train a doc2vec based on user browsing history (urls tagged to user_id). I use chainer deep learning framework. There are more than 20 millions (user_id and urls) of embeddings to ...
Aljo Jose's user avatar
  • 171
2 votes
0 answers
134 views

how Can we add extra word embedding to the pytorch funnel transformer?

i was approaching NLP sequence classification problem (3 classes) using huggingface transformers (funnel-transformer/large) and tensorflow. first i created laserembedding like this : ...
Syed Mobassir's user avatar
2 votes
2 answers
5k views

How to calculate the mean average of word embedding and then compare strings using sklearn.metrics.pairwise

I am totally new to this topic, that's why I am so confused or stuck in this code for a while, but I am not sure how to solve it correctly. My goal is to write a short text embedding using vector ...
test's user avatar
  • 21
2 votes
0 answers
174 views

Ways to cluster word senses with word embeddings

I'm trying to semantically cluster polysemous words or word with different meanings in a corpus for my class study and I want to do it by word embeddings but I have no Idea how to reach to the ...
amkyp's user avatar
  • 21
2 votes
1 answer
34 views

Types of averages when analyzing sentences

I have a list of words and their frequencies in a text corpus. So there are words like "a", "what", "some" that have really high frequencies, and other like "...
johnnydoe's user avatar
  • 141
2 votes
0 answers
766 views

Learning words embedding for bigrams and unigrams in a corpus

I am working on a topic modeling for tweets projects. I have generated my topics using both unigrams and bigrams. Topics are defined with a mixture of both bigrams and unigrams. Now I am planning to ...
Espoir Murhabazi's user avatar
2 votes
0 answers
315 views

How to use paraphrase_mining using sentence transformers pre-trained model

I am trying to find similarity between sentences using a pre-trained sentence-transformers model. I am trying to follow the code here - https://www.sbert.net/docs/usage/paraphrase_mining.html In trial ...
Regressor's user avatar
  • 121
2 votes
1 answer
30 views

Training pipelines where featurization/NLP is more expensive than backprop

I'm working on a document classification project and I'm using a neural net in tensorflow, where the features are 300-dimensional word embeddings, either from fastext or word2vec (yes I know that ...
generic_user's user avatar
2 votes
0 answers
106 views

More pre-trained embeddings for PyTorch Big Graph

Apart from the entire wikidata, are there any other PyTorch Big Graph pre-trained graph embeddings on smaller sized knowledge graph, like freebase-15k? I do not have the resources to build it from ...
user209139's user avatar
2 votes
2 answers
219 views

Semantic Search

There is a problem we are trying to solve where we want to do semantic search on our set of data, i.e we have a domain specific data (example: sentences talking about automobiles) Our data is just a ...
Farhaan Bukhsh's user avatar
2 votes
1 answer
149 views

Building own embedding from sequence

I have 100 sequences of the word (i.e., action for completing a task). Each of the sequences contains around 350 actions(115 unique actions but all the actions are not used in each sequence. Some of ...
Bloodstone Programmer's user avatar
2 votes
0 answers
85 views

NLP based Data Preprocessing Method to Improve Disease Name Prediction Using CRF and Word Embedding

I built a model( using CRF along bi lstm) to Predict New Disease Name/Entities from medical text data but the problem is Disease name appears only 5,6 times in 1 text file but on average text file ...
Programmer11's user avatar
2 votes
1 answer
396 views

Predicting word from a set of words

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
Oren Matar's user avatar
2 votes
1 answer
646 views

What are the equations involved in calculation of the parameters of embedding layer?

I'm trying to perform sentiment analysis on some data using keras.I'm using embedding layer and then LSTM. I know that embedding layer decreases the sparsity of the one hot encodings of the words and ...
Harish Reddy's user avatar
2 votes
1 answer
260 views

Improve results using user input

I've developed a tool that retrieve the closest expressions from a database based on what the user typed. (using word embedding - a comparison is made between each expression from the database and the ...
Martin's user avatar
  • 21
2 votes
0 answers
504 views

Keras value error: Operands could not be broadcast with with shapes(100,100) - GRU

I am trying to use Hierarchical Attention Networks for classification of news articles using 20 newsgroup dataset that i downloaded from the internet. I came across this code of the implementation and ...
Shivam...'s user avatar
  • 123
2 votes
2 answers
2k views

Combining different features as input to Neural Network

I use two different sources of information as input to my neural model. The model takes a word as input and produces binary [1/0] output. I have represented each word by using its word embedding (1024 ...
zwlayer's user avatar
  • 269
2 votes
1 answer
2k views

Enable Mini-batch Processing on PyTorch Word Embeddings

I am new to PyTorch and trying to create word embeddings. I started with the example below and everything works fine and it completes relatively quickly. ...
Skiddles's user avatar
  • 988
2 votes
0 answers
206 views

NER at sentence level or document level?

Should NER models (LSTM or CRF) take input training data at sentence level or paragraph level? Let's say we have this input text, and we would like to do Named Entity Extraction: GOP Sen. Rand ...
Franklin Dong's user avatar
1 vote
0 answers
20 views

word2vec predicts the same word for all inputs

i build word2vec network with 2 linear layers from pytorch. for every word as an input i consistently train model to predict words before and after, for example: i was visiting my grandma's house, for ...
Тима 's user avatar
1 vote
0 answers
44 views

Is openAI text generation models an extension of embedding models?

we can creating embeddings using below code ...
Vinay Sharma's user avatar
1 vote
0 answers
34 views

"text" parameter in pinecone call from langchain

In this tutorial, I do not understand what "text" refers to vectorstore = Pinecone(index, embeddings.embed_query, "text") Could you please help?...
Karl 17302's user avatar
1 vote
0 answers
54 views

I can't get good performance from BERT

I trained NLP models. This is a subset (200 instances) of my data set of 10,000 instances:This the link of the dataset on pastebin I compare an LSTM model with a glove model and a BERT model. I ...
Seydou GORO's user avatar
1 vote
0 answers
142 views

Otimization of similarity search for multiple embeddings by creating a weighted artificial embedding

I have embeddings of text created with a BERT model. A group of these embeddings should be used to find similar embeddings corresponding to this group. I know that you can use average or max (or ...
soph's user avatar
  • 113
1 vote
1 answer
89 views

Which pre-trained model to select to generate embeddings from shop names written in English?

Good afternoon! I have a dataset with thousands of shop names written in English. Several shop names might belong to one business entity, for instance, shops with names "KFC 001", "WWW....
rsx's user avatar
  • 11
1 vote
0 answers
100 views

Is normalization of word embeddings important?

I am doing actor-critic reinforcement learning for an environment that is best represented as a "bag-of-words". For this reason, I have opted to use a single body, multi-head approach for ...
Ryan Keathley's user avatar
1 vote
0 answers
86 views

Machine learning with mixed variables dataset (numerical, categorical and embeddings)

I'm working on a machine learning project where I'm trying to predict the revenue of a movie. My dataset contains mixed data types. There are numerical features (rating, number of votes, release year,....
Mathieu Rousseau's user avatar
1 vote
1 answer
103 views

weights of coocurrence matrix in glove

I was studying the theory behind glove and was checking out some implementations of it. Before passing the data to its neural networks, I noticed that the weights of the co-occurrence matrix aren't ...
mihael's user avatar
  • 121
1 vote
0 answers
491 views

What are the inputs of encoder and decoder layers of transformer architecture?

In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the ...
canP's user avatar
  • 101
1 vote
0 answers
42 views

Which model is better able to understand the difference that two sentences are talking about different things?

I'm currently working on the task of measuring semantic proximity between sentences. I use fasttext train _unsiupervised (skipgram) for this. I extract the sentence embeddings and then measure the ...
Ir8_mind's user avatar
  • 183
1 vote
0 answers
51 views

Contextual word embeddings from pretrained word2vec vectors

I would like to create word embeddings that take context into account, so the vector of the word Jaguar [animal] would be different from the word Jaguar [car brand]. As you know, word2vec only gives ...
amks1212's user avatar
1 vote
0 answers
320 views

How are the embedding and context matrices created and updated in word embedding?

I am struggling to understand how word embedding works, especially how the embedding matrix $W$ and context matrix $W'$ are created/updated. I understand that in the Input we may have a one-hot ...
Revolucion for Monica's user avatar
1 vote
0 answers
54 views

How to deal with words out of the vocabulary CBOW implementation

I'm studying word2vec theory, and I decided to implement the Continuous Bag of Words model from zero. I know the primary pipeline for this: Preprocess a corpus: remove stopwords, lemmatization, etc. ...
mihael's user avatar
  • 121
1 vote
1 answer
527 views

What are different ways to reduce size of word2vec vectors file?

I am working on an application with memory constraints. We are getting vectors from python Gensim models but need to transmit copies of them to react native mobile app and potentially in-browser JS. ...
Aditya Jain's user avatar
1 vote
1 answer
263 views

How can I use Wikipedia2vec model for embedding my article named entities as 40% entities are not in a wikipedia?

I have news articles in my dataset containing named entities. I want to use the Wikipedia2vec model to encode the article's named entities. But some of the entities (around 40%) from our dataset ...
sajankar9's user avatar
1 vote
0 answers
13 views

Bad accuracy on predicting user ratings upon his reviews

I am trying to predict user ratings based on books he rated(ratings range between 1-10). So I encoded the summary of the books and then tried to train the model with the text encoding(practically the ...
Greekatos's user avatar
1 vote
0 answers
192 views

Should I use Pad Sequence when using Word Vectors?

I have an unbalanced text data set. I want to use word vectors to embed words. When I use pad sequence? Before or after the word vector? I tried it, after the word vector I used pad sequence but my ...
grace's user avatar
  • 13
1 vote
0 answers
36 views

GloVe dot product optimized for non-comutative data whilst the operation itself being commutative

To my current knowledge, GloVe word vectors dot product are optimized to be the w_i ⋅ w_j = log⁡(P(ⅈ|j)) The probability being computed from a cooccurance matrix. However, dot product is a commutative ...
Arik's user avatar
  • 23
1 vote
0 answers
150 views

Word embedding autoencoder

I'm trying to train a word embedding autoencoder, but it either doesn't train, or trains but doesn't make predictions. I know I'm doing something wrong, so any help is greatly appreciated. Here is my ...
user1513335's user avatar
1 vote
0 answers
136 views

Plotting cosine similarities in 3d space from word embeddings

I'm working on a project in which I want to estimate biases from a large corpus of newspaper articles using word2vec. Following this and this paper, biases are calculated by constructing dimension x ...
fritsvegters's user avatar
1 vote
1 answer
2k views

Not clear about relative position bias

I've been reading the Swin Transformer paper and came across relative position bias concept. I'm not able to figure out how is it more effective than positional embeddings. I hope someone can explain ...
pramesh's user avatar
  • 141
1 vote
1 answer
152 views

Ensure trained word embeddings get high similarity with particular words

I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times ...
Clock Slave's user avatar