Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
0
votes
1answer
7 views

BERT is running out of memory in forward pass for my dictionary

Running code from this answer, my BERT is running out for my 4k words dictionary. I don't need to do anything with these words yet, just make embeddings for my data. So, using this exactly: ...
0
votes
0answers
8 views

Why does word2vec try to maximize the dot product of the center word vector and context?

I am learning about the maths behind word2vec from this tutorial. u are the embeddings for the center word and v for the context. It appears that this dot product is to be maximized. Why the context ...
0
votes
2answers
32 views

How to JUST represent words as embeddings by pretrained BERT?

I don't have enough data (i.e. I don't have enough texts) --- have only around 4k words in my dictionary. I need to compare given words, then I need to representate it as embedding. After the ...
1
vote
3answers
42 views

Building a multiclass classifier that can handle classes it has never seen?

I am given a dataset that has free-form text and a category associated with it. There are 100 different categories and 3000 records for each category. The goal is build a multiclass classification ...
0
votes
0answers
41 views

Cluster images labels in some given categories using word embeddings

Given: set of images Labels in string format each one. Also I've given a set of Categories, also in string. ($Images \neq Categories $) Goal: I need to map given labels to given categories to "...
2
votes
3answers
139 views

The reason behind using a pre-trained model?

These last month I have been studying all about word embeddings and the most known pre-trained word embeddings, Word2Vec, GloVe, FastText, etc. I have read many times how important It is to take ...
0
votes
1answer
20 views

Word2Vec: Identifying many-to-one relationships between words

Standard introductory examples in Word2Vec, like king - queen = man - woman and tokyo - japan = london - uk, involve one-to-one ...
0
votes
0answers
8 views

How to train effective domain specific word2vec models?

I'm training a word2vec model based on our custom data. Due to the size of data and the resources available to me, I'll use pyspark to train spark's built-in word2vec model. However, I'm curious to ...
-1
votes
0answers
9 views

Compare my word embedding models (Count based, PMI, SPPMI)

I am going to build some models over wiki-dump dataset and then try to compare the results to WS353 (for word similarity). So, I need to check whether my understanding is correct or not. Firstly, I ...
1
vote
0answers
10 views

When are subword ngrams trained in fasttext? (Enriching Word Vectors with Subword Information)

when is the training for subword ngrams done? is it done simultaneously as when the word representation are trained? or is it done at the end, after word representations are created? fasttext ...
0
votes
0answers
44 views

Difference between NCE-Loss and InfoNCE-Loss

I started looking into word2vec and was wondering what the connection/difference between the NCE-Loss and the infoNCE-Loss is. I get the basic idea of both. I have a hard time deriving one from ...
0
votes
1answer
66 views

Sentence embeddings with LSTM to classify the sentences is not working

I am trying to build LSTM NN to classify the sentences. I have seen many examples where sentences are converted to word vectors using glove, word2Vec and so on here is an example of it. This solution ...
1
vote
0answers
13 views

How can I convert my predictions to text after predicting using RNN?

I'm building PoS tagger for our language. I give tokens to the words and tags using Tokenizer(). Functions for word and tag are different. ...
0
votes
1answer
105 views

Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?

I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it ...
2
votes
0answers
35 views

The embeddings of GoogleNews-vectors-negative300.bin is stemmed or not?

I will use the embedding GoogleNews-vectors-negative300.bin data to do some project. Just want to know whether it was stemmed of the words?
1
vote
1answer
27 views

Preparing training data for NLP machine learning task

I have the natural language sentences as follows: This is a black chair. It is next to the table. Each phrase that represents an object is annotated with an object ...
0
votes
1answer
37 views

Word embedding for a single word

I want to tackle email similarity with a word embedding approach, not an expert in embedding text but it is possible to embed emails where similar emails have similar vectors? is that okay?. I know ...
4
votes
0answers
46 views

Can we use embeddings or latent vectors for a recommender system?

I'm having a hard time understanding why people use any vector they find as a candidate for a recommender system. In my mind, a recommender system requires a space where distance represents similarity....
0
votes
0answers
13 views

Skipgram model theory confusion

In the output layer of a skipgram model, there are $|\text{Context}|*|\text{Vocab}|$ values. And for each context word, the values are basically the dot product of the input word representation and ...
2
votes
0answers
23 views

Medical NER for French language

I'm currently exploring the options to extract medical NER specifically for French language. I tried SpaCy's general French NER but it wasn't helpful to the cause (...
1
vote
0answers
28 views

Self-supervised learning for automatic labeling of data using LDA and Word2Vec

I am trying to implement this paper A Brand-New Look at You: Predicting Brand Personality in Social Media Networks with Machine Learning for labeling Twitter data of brands with a corresponding brand ...
0
votes
0answers
49 views

How does Google's Universal Sentence Encoder deal with out-of-vocabulary terms?

It seems to output embeddings even for random jibberish, and the similarity is even high for this particular pair of jibberish. ...
1
vote
0answers
38 views

Learning words embedding for bigrams and unigrams in a corpus

I am working on a topic modeling for tweets projects. I have generated my topics using both unigrams and bigrams. Topics are defined with a mixture of both bigrams and unigrams. Now I am planning to ...
0
votes
0answers
11 views

What should be the value of “Embedding_initializer” flag while fine tuning pretrained embeddings

Need help to resolve my understanding of the Embedding layer parameters from keras. As per my understanding, providing an explicit embedding matrix as weight parameter and setting trainable flag to ...
0
votes
0answers
29 views

How to determine sentence similarity labels for sentence transformer fine-tuning?

I'm using the Sentence Transformer library to fine-tune pre-trained transformer models. In the fine tuning documentation, the example provided requires labels (from 0 to 1) that indicate the ...
1
vote
1answer
24 views

What is a good approach for embedding both textual and spatial features for document classification?

I am working on a document classifier that can perform the classification based on the document structure as well. My plan is to get the word embedding as well as the word coordinates and somehow ...
1
vote
1answer
28 views

How to find coherence between a large number of sentences

I have a list of sentences returned as a result of a document search algorithm. I want to determine if the results returned are semantically close/similar/coherent using some sort of metric. For a ...
0
votes
0answers
27 views

How to interpret feature importance in text classification using Fasttext?

Once the text is converted into a vector of size(1,100), how can we interpret and backtrace a word's importance which helped in classification?
3
votes
3answers
136 views

Dot product for similarity in word to vector computation in NLP

In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word. Uo is word vector for outside word Vc is word ...
0
votes
0answers
11 views

GloVe word embeddings containing sentiment?

I've been researching sentiment analysis with word embeddings. I read papers that state that word embeddings ignore sentiment information of the words in the text. One paper states that among the top ...
0
votes
0answers
29 views

Adding extra feature to word2vec embeddings for use in classification

I have a word2vec model that will transform my text into vectors, and I need to use them to classify the text. My data is essentially a timeseries of chat messages, so I think that the message ...
4
votes
2answers
187 views

how to run bert's pretrained model word embeddings faster?

I'm trying to get word embeddings for clinical data using microsoft/pubmedbert. I have 3.6 million text rows. Converting texts to vectors for 10k rows takes around 30 minutes. So for 3.6 million rows, ...
3
votes
1answer
64 views

Word2Vec: Why do some dimensions of an embedding have an interpretation, and why does addition/subtraction of embedding vectors work?

I'm reading about Word2Vec from this source: http://jalammar.github.io/illustrated-word2vec/. Below is the heatmap of the embeddings for various words. In the source, it's claimed that we can get an ...
0
votes
1answer
57 views

Context Based Embeddings vs character based embeddings vs word based embeddings

I am working on a problem that uses English alphabets in the text but the language is not English. Its a mixture of English and different language text. But all words are written using English ...
0
votes
0answers
17 views

Impact of a new word on word embedding vectors

Question What is the impact of a new word on the word embedding vectors already trained before the word is invented? For instance, at November 2019, there existed multiple pre-trained models from ...
1
vote
0answers
8 views

Very infrequent values in embedding layers

I have a categorical input that is very imbalanced. 90% of the values are either A or B and frequencies for C, D, E, F, etc are as little as 1. I am using an embedding layer for this input and the ...
0
votes
1answer
58 views

Continuous Bag Of Words (CBOW) network architecture?

Looking into word2vec like embeddings I found this exercise on PyTorch's website which prompts the reader to implement a CBOW network in PyTorch. My question is about the architecture to implement ...
0
votes
0answers
7 views

Projection layer function

I am trying to understand word-vectors and was reading this paper https://arxiv.org/pdf/1301.3781.pdf This paper proposes CBOW architecture which uses projection layer. What is projection layer? I ...
0
votes
0answers
75 views

Applying word embedding function on a large pandas dataset

We have a pandas dataframe with one column ('message') and 3.9 million rows and need to convert these messages to their word embeddings using Google's Universal Sentence encoder. ...
1
vote
0answers
32 views

How to test the quality of a word embedding?

I have trained a word2vec model using GenSim 4. The problem is that my corpus is quite small. How can I test the quality of the word embeddings I have obtained? Is there some standard measures to do ...
0
votes
0answers
42 views

Using bert (or fitbert) for predicting masked words from word candidates

Fitbert (which is based on Bert) can be used to predict (fill in) a masked word from a list of candidates as below: ...
0
votes
0answers
42 views

Word2Vec vs LexVec vs GloVe

I'm on a NLP project and found a resource that has the three word representations mentioned in the question name, and I am struggling to find one place where they all are explained and compared. As I ...
0
votes
1answer
20 views

Vector representation of documents for text classification

I'm looking for proper method of document embeddings. I know that doc2vec will give me the vector representations for given corpus, but how do I embed new documents? I need to train neural network ...
2
votes
1answer
883 views

How pre-trained BERT model generates word embeddings for out of vocabulary words?

Currently, I am reading BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I want to understand how pre-trained BERT generates word embeddings for out of vocabulary ...
0
votes
0answers
27 views

Math behind word2vec in neural network

I built a neural network for my NLP preblem using GloVe and an embedding layer. Each word is converted to a vector of 100 dimensions and input_length is 300. Word2Vec has 68,546 words. How does the ...
1
vote
0answers
28 views

How to use paraphrase_mining using sentence transformers pre-trained model

I am trying to find similarity between sentences using a pre-trained sentence-transformers model. I am trying to follow the code here - https://www.sbert.net/docs/usage/paraphrase_mining.html In trial ...
1
vote
0answers
22 views

What is the structure and dimension of input passed to neural network when training CBOW and SKIP GRAM word embedding

I am confused about input passed to neural network in natural language processing (NLP) when training CBOW word embedding from scratch. I read the paper and have ...
1
vote
1answer
42 views

DBSCAN on textual and numerical columns

I have a dataset which has two columns: title price sentence1 12 sentence2 13 I have used doc2vec to convert the ...
2
votes
1answer
23 views

Does the output of the Sequence-to-Sequence encoder model exist in the same semantic space as the inputs (Word2vec)? [closed]

Does the output generated from the LSTM encoder module exist in the same semantic space as the original word vectors? If so, say for example we have a sentence and we pass it through the encoder to ...
2
votes
1answer
203 views

How to choose dimension of Keras embedding layer?

Looking for some guidelines to choose dimension of Keras word embedding layer. For example in a simplified movie review classification code: ...

1
2 3 4 5
8