Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
0 votes
1 answer
27 views

Range of values of BERT and other embeddings?

Are the values in all NLP models' embeddings between the range -1 to 1? If not, what models use a different range (or decimal points)? And what could be the reason for that shift/change?
user avatar
  • 103
0 votes
0 answers
22 views

Using BERT embeddings as input for transformer architecture

I will use BERT's embedding weights (as discussed here) for embedding in embedding layers of the transformer model. But my question is: don't embeddings of BERT already go through the whole encoding ...
user avatar
  • 45
0 votes
0 answers
17 views

Transfer Learning transformer architecture

I would like to make transfer learning for transformer architecture. The input of the encoder and decoder must be word embeddings. So I wanted to use a pre-trained BERT model as a word embedder and ...
user avatar
  • 45
0 votes
0 answers
17 views

What are the inputs of encoder and decoder layers of transformer architecture?

In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the ...
user avatar
  • 45
0 votes
0 answers
12 views

Fakebert implementaion

I am trying to implement this architecture of fake bert for fake news detection, but I don't know how to feed the word embedding from Bert. Help, please.
user avatar
0 votes
0 answers
11 views

How can I use the embedding generated by mBERT with a CNN or SVM as a classifier?

I have a school project and need to use the embeddings generated by BERT, for example, mBERT, and using a classifier like SVM, CNN... Any help, please. Thank you!
user avatar
1 vote
0 answers
37 views

Which model is better able to understand the difference that two sentences are talking about different things?

I'm currently working on the task of measuring semantic proximity between sentences. I use fasttext train _unsiupervised (skipgram) for this. I extract the sentence embeddings and then measure the ...
user avatar
  • 143
0 votes
0 answers
11 views

Can I get un-normalized vectors from the TF USE model?

I'm using this Universal Sentence Encoder (USE) model to get embeddings of a set of texts, each text corresponding to a newspaper article. In order to build a Recommender System, I generate user ...
user avatar
1 vote
0 answers
13 views

Contextual word embeddings from pretrained word2vec vectors

I would like to create word embeddings that take context into account, so the vector of the word Jaguar [animal] would be different from the word Jaguar [car brand]. As you know, word2vec only gives ...
user avatar
1 vote
1 answer
26 views

Is there a sensible notion of 'character embeddings'?

There are several popular word embeddings available (e.g., Fasttext and GloVe); In short, those embeddings are a tool to encode words along with a sensible notion of semantics attached to those words (...
user avatar
1 vote
0 answers
12 views

How are the embedding and context matrices created and updated in word embedding?

I am struggling to understand how word embedding works, especially how the embedding matrix $W$ and context matrix $W'$ are created/updated. I understand that in the Input we may have a one-hot ...
user avatar
5 votes
1 answer
212 views

Sum vs mean of word-embeddings for sentence similarity

So, say I have the following sentences ["The dog says woof", "a king leads the country", "an apple is red"] I can embed each word using an ...
user avatar
1 vote
1 answer
37 views

Why is Word2vec regarded as a neural embedding?

In the skip-gram model, the probability that a word $w$ is part of the set of context words $\{w_o^{(i)}\}$ $(i= 1:m)$ where $m$ is the context window around the central word, is given by: $$p(w_o | ...
user avatar
  • 43
1 vote
1 answer
38 views

Why we need to 'train word2vec' when word2vec itself is said to be 'pretrained'?

I get really confused on why we need to 'train word2vec' when word2vec itself is said to be 'pretrained'? I searched for word2vec pretrained embedding, thinking i can get a mapping table directly ...
user avatar
  • 359
1 vote
1 answer
10 views

Application of bag-of-ngrams in feature engineering of texts

I've got few questions about the application of bag-of-ngrams in feature engineering of texts: How to (or can we?) perform word2vec on bag-of-ngrams? As the feature space of bag of n-gram increases ...
user avatar
  • 359
1 vote
1 answer
87 views

Do we perform text embedding before or after train-test splitting?

Do we perform text embedding before or after train-test splitting? I know that for encoding variables, usually done after the split. However, I'm not sure if that's also the case for text processing?
user avatar
  • 359
1 vote
0 answers
16 views

How to deal with words out of the vocabulary CBOW implementation

I'm studying word2vec theory, and I decided to implement the Continuous Bag of Words model from zero. I know the primary pipeline for this: Preprocess a corpus: remove stopwords, lemmatization, etc. ...
user avatar
  • 11
1 vote
1 answer
163 views

Measuring similarity from massive embedded vectors

I am given a set of 10,000 journal articles, with their corresponding 100th-dimension embedded vectors. (The way they are embedded is unknown, but I'm guessing it is ...
user avatar
  • 13
0 votes
0 answers
15 views

How to choose between Genism Word2Vec and Keras embedding?

I've seen this post on the difference between keras embedding and word2vec in Genism. It gives me the impression that Word2Vec in Genism is kinda pre-trained word vectors. I wish very much the ...
user avatar
  • 359
0 votes
0 answers
9 views

The output of CBOW, compared to Skipgram

From my undertanding the desired outputs from skipgram is actually the word embedding for a word, as pointed in red in the picture. But how about CBOW? Is the goal of CBOW training also aim at the ...
user avatar
  • 359
0 votes
0 answers
14 views

Weighting Sentence Similarity by salience or frequency

It seems like the new standard in text search is sentence or document similarity, using things like BERT sentence embeddings. However, these don't really have a way to consider the salience of ...
user avatar
1 vote
1 answer
66 views

What is the meaning of two embedding layers in a row?

I've noticed in one deep pre-trained textual neural network that there are two embedding layers in the beginning and I don't quite understand why there are two of them. As far as I understand (correct ...
user avatar
  • 157
0 votes
0 answers
17 views

Is it possible to resize or compress word embeddings?

I have a 600 dimension GloVe word embedding with me, pretrained on a set of documents which suits my use case. However, I would like to reduce the number of embeddings to 200 dimensions. Retraining ...
user avatar
1 vote
1 answer
49 views

What are different ways to reduce size of word2vec vectors file?

I am working on an application with memory constraints. We are getting vectors from python Gensim models but need to transmit copies of them to react native mobile app and potentially in-browser JS. ...
user avatar
0 votes
0 answers
15 views

Learning an embedding for multiple types of text features

I have a problem where my goal is to learn a representation / embedding for various text features. In the initial formulation my dataset looks like the following: ...
user avatar
  • 161
1 vote
1 answer
143 views

What are the exact differences between Word Embedding and Word Vectorization?

I am learning NLP. I have tried to figure out the exact difference between Word Embedding and Word Vectorization. However, seems like some articles use these words interchangeably. But I think there ...
user avatar
  • 13
0 votes
0 answers
29 views

how can i average subword embedding?

how can i average subword embedding vectors to generate an approximate vector for the original word as i get the embedding using this function ...
user avatar
  • 15
1 vote
1 answer
33 views

Latent space vs Embedding space | Are they same?

I am going through variational autoencoders and it is mentioned that: continuity (two close points in the latent space should not give two completely different contents once decoded) and completeness ...
user avatar
  • 135
2 votes
3 answers
63 views

Best way to vectorise names and addresses for similarity searching?

I have a large dataset of around 9 million people with names and addresses. Given quirks of the process used to get the data it is highly likely that a person is in the dataset more than once, with ...
user avatar
  • 187
0 votes
3 answers
59 views

Does word2vec fail for window size equal to sentence size

Will word2vec fail if sentences contain only similar words, or in other words, if the window size is equal to the sentence size? I suppose this question boils down to whether word to vec considers ...
user avatar
0 votes
1 answer
15 views

How do companies handle changing natural language

I am assuming large social medias like Twitter handle hashtags using some sort of embedding, so that similar tweets can be found or suggested. Maybe this is a bad assumption- maybe someone can clarify....
user avatar
0 votes
2 answers
124 views

Word Embedding Dimensions Reduction

In my NLP task, I use Glove to get each word embedding, Glove gives 50 float numbers as an embedding for every word in my sentence, my corpus is large, and the resulted model is also large to fit my ...
user avatar
  • 13
0 votes
1 answer
115 views

How to compute sentence embedding from word2vec model?

I am new to NLP and I'm trying to perform embedding for a clustering problem. I have created the word2vec model using Python's gensim library, but I am wondering ...
user avatar
0 votes
0 answers
38 views

What is the difference between fine-tuning and retraining for the pre-trained word embedding model such as Glove?

I have two questions: What is the difference between fine-tuning and retraining for the pre-trained word embedding model such as Glove, and which is the best for a specific domain? Do I need to ...
user avatar
0 votes
0 answers
14 views

How different are the word embeddings trained from Skipgram and CBOW?

Since what we are interested about usually from CBOW and Skipgram are the by-product word embeddings from the networks, how does the word embeddings they produce differ? When to use which to get the ...
user avatar
  • 359
0 votes
0 answers
14 views

Can word2vec's neural network itself, and not the embedding weights, be used for word prediction?

Given the shallow neural network that was used to train e.g.: a skip-gram model, my question is: Can we actually use this network to predict probable context words? What is the output of this network? ...
user avatar
0 votes
0 answers
11 views

Can linguistic relationships be derived from word-embedding models?

Transformations in Embedding space One fascinating property of word-embedding models like GloVe and word2vec is that linguistic transformations can be described as bi-directional vectors mapping ...
user avatar
0 votes
1 answer
156 views

Positional encoding without input embedding

Does it make sense to use a positional encoding in attention when the input tokens do not go through an embedding layer? In NLP models, the embedding maps a word to real numbers. ...
user avatar
0 votes
1 answer
303 views

What are the differences between bert embedding and flair embedding

I read about BERT embedding model and FLAIR embedding model, and I'm not sure I can tell what are the differences between them ? ...
user avatar
0 votes
0 answers
30 views

mismatch between shapes using BERT

I used PyTorch to get word-embeddings using BERT for my sentences which are 150074 sentences....
user avatar
0 votes
2 answers
40 views

zero padding problem [closed]

i need to implement this code by using padding ...
user avatar
0 votes
2 answers
170 views

Why does averaging word embedding vectors (exctracted from the NN embedding layer) work to represent sentences?

I'm puzzling to understand why the method of averaging word embeddings works in order to obtain sentence embedding, in particular considering the exercize of this post How to obtain vector ...
user avatar
0 votes
0 answers
52 views

Conceptual question about cosine similarity and clustering algorithms for word embeddings

Is the following statement true? https://stats.stackexchange.com/q/256778 The value of cosine similarity between two terms itself is not indicator whether they are similar or not. If yes then how is ...
user avatar
1 vote
1 answer
29 views

does ValueError: 'rat' is not in list means not exist in tokenizer

Does this error means that the word doesn't exist in the tokenizer return sent.split(" ").index(word) ValueError: 'rat' is not in list the code sequences ...
user avatar
0 votes
1 answer
318 views

How can i get the vector of word using BERT?

I need to get word-vectors using BERT and got this function that i think it should be the one i need ...
user avatar
2 votes
1 answer
320 views

Custom Named-Entity Recognition (NER) in product titles using deep learning

I am new to machine learning and Natural Language Processing (NLP). I am trying to identify which brand, product name, dimension, color, ... a product has from its product title. That is, from 'Sony ...
user avatar
0 votes
1 answer
353 views

How to calculate the mean average of word embedding and then compare strings using sklearn.metrics.pairwise

I am totally new to this topic, that's why I am so confused or stuck in this code for a while, but I am not sure how to solve it correctly. My goal is to write a short text embedding using vector ...
user avatar
  • 1
0 votes
0 answers
45 views

How can I use Wikipedia2vec model for embedding my article named entities as 40% entities are not in a wikipedia?

I have news articles in my dataset containing named entities. I want to use the Wikipedia2vec model to encode the article's named entities. But some of the entities (around 40%) from our dataset ...
user avatar
3 votes
1 answer
522 views

Which method is more suitable? overfitting of traning data or low accuracy?

Recently, I tested two methods after embedding in my data, using Keras. Convolution after embedding Maxpooling after embedding The first method's loss and validation loss are like, The second ...
user avatar
3 votes
1 answer
91 views

Question on embedding similarity / nearest neighbor methods [SCANN Paper]

Question on embedding similarity / nearest neighbor methods: In https://arxiv.org/abs/2112.04426 the DeepMind team writes: For a database of T elements, we can query the approximate nearest neighbors ...
user avatar
  • 2,400

1
2 3 4 5
9