Questions tagged [word-embeddings]

Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

Conceptual question about cosine similarity and clustering algorithms for word embeddings

Is the following statement true? https://stats.stackexchange.com/q/256778 The value of cosine similarity between two terms itself is not indicator whether they are similar or not. If yes then how is ...
1
vote
1answer
17 views

does ValueError: 'rat' is not in list means not exist in tokenizer

Does this error means that the word doesn't exist in the tokenizer return sent.split(" ").index(word) ValueError: 'rat' is not in list the code sequences ...
0
votes
1answer
44 views

How can i get the vector of word using BERT?

I need to get word-vectors using BERT and got this function that i think it should be the one i need ...
0
votes
0answers
18 views
2
votes
1answer
36 views

Custom Named-Entity Recognition (NER) in product titles using deep learning

I am new to machine learning and Natural Language Processing (NLP). I am trying to identify which brand, product name, dimension, color, ... a product has from its product title. That is, from 'Sony ...
0
votes
1answer
12 views

How to calculate the mean average of word embedding and then compare strings using sklearn.metrics.pairwise

I am totally new to this topic, that's why I am so confused or stuck in this code for a while, but I am not sure how to solve it correctly. My goal is to write a short text embedding using vector ...
0
votes
0answers
7 views

How can I use Wikipedia2vec model for embedding my article named entities as 40% entities are not in a wikipedia?

I have news articles in my dataset containing named entities. I want to use the Wikipedia2vec model to encode the article's named entities. But some of the entities (around 40%) from our dataset ...
3
votes
1answer
496 views

Which method is more suitable? overfitting of traning data or low accuracy?

Recently, I tested two methods after embedding in my data, using Keras. Convolution after embedding Maxpooling after embedding The first method's loss and validation loss are like, The second ...
2
votes
1answer
48 views

Question on embedding similarity / nearest neighbor methods [SCANN Paper]

Question on embedding similarity / nearest neighbor methods: In https://arxiv.org/abs/2112.04426 the DeepMind team writes: For a database of T elements, we can query the approximate nearest neighbors ...
0
votes
0answers
7 views

Word-level text generation with word embeddings – outputting a word vector instead of a probability distribution

I am currently researching the topic of text generation for my university project. I decided (ofc) to go with a RNN getting a sequence of tokens as input with a target of predicting the next token ...
0
votes
0answers
9 views

Are the final output or the output of each layers in transformer an embedding for input words?

In transformers, it seems for each input word there is an output at each layer of transformer and they are flow in parallel. I guess the output of each layer is a representation for the input word, ...
1
vote
1answer
20 views

When to use GloVe vocabulary vs. building a vocabulary from the training data?

While studying some (pytorch) examples that use pretrained GloVe vectors I came across two variants: Use the vocabulary of the GloVe vectors and thus initialize the embedding layer with the ...
1
vote
0answers
11 views

Bad accuracy on predicting user ratings upon his reviews

I am trying to predict user ratings based on books he rated(ratings range between 1-10). So I encoded the summary of the books and then tried to train the model with the text encoding(practically the ...
1
vote
0answers
18 views

Should I use Pad Sequence when using Word Vectors?

I have an unbalanced text data set. I want to use word vectors to embed words. When I use pad sequence? Before or after the word vector? I tried it, after the word vector I used pad sequence but my ...
1
vote
1answer
30 views

How to get word embedding in CBOW?

I find clear explanations for skip-gram model. We take the output weight matrix, multiply it with the one-hot vector of the word we want to get the embedding. How does it work in case of CBOW? I know ...
1
vote
0answers
12 views

GloVe dot product optimized for non-comutative data whilst the operation itself being commutative

To my current knowledge, GloVe word vectors dot product are optimized to be the w_i ⋅ w_j = log⁡(P(ⅈ|j)) The probability being computed from a cooccurance matrix. However, dot product is a commutative ...
0
votes
0answers
6 views

In the node2vec model derivation, what does it mean for node representations to be "Symmetric in Feature Space"?

The main derivation of the probabilistic model in Node2Vec goes as follows (paper available on ArXiv: https://arxiv.org/pdf/1607.00653.pdf): We formulate feature learning in networks as a maximum ...
1
vote
2answers
151 views

How to obtain vector representation of phrases using the embedding layer and do PCA with it

I am trying to understand from both a conceptual and a Python code point of view, how to represent phrases that are present in a corpus (that is used to train a neural network to classify phrases) as ...
2
votes
2answers
40 views

How to examine if a Doc2Vec model is sufficiently trained?

I started experimenting with gensim's Doc2Vec for sentiment analysis. For the training of the embedding itself, I have seen examples using a reduced learning rate with a few 10s or even a few hundred ...
2
votes
0answers
63 views

Ways to cluster word senses with word embeddings

I'm trying to semantically cluster polysemous words or word with different meanings in a corpus for my class study and I want to do it by word embeddings but I have no Idea how to reach to the ...
2
votes
1answer
39 views

Using word embeddings as features in classification algorithms?

I see there are ways to combine word vectors to form documents by taking averages or weighted averages. However, as a result of averaging there is a loss of information. Are there ways to retain the ...
0
votes
0answers
10 views

Training fasttext on your own corpus

I want to train fasttext on my own corpus. However, I have a small question before continuing. Do I need each sentences as a different item in corpus or can I have many sentences as one item? For ...
0
votes
0answers
30 views

How to build vocabulary file for NLP embeddings efficiently?

I am currently building various word embeddings for my NLP project, ranging from Word2Vec, ELMo, LINE etc. I am looking to train ELMo using AllenNLP, a Python package for NLP, using the tutorial here. ...
0
votes
0answers
33 views

GloVe Embedding Matrix "could not broadcast input array from shape (0) into shape (300)"

I'm working on Quora Question Pairs data set. I'm trying to get embedding matrix for GloVe with the following code: ...
0
votes
0answers
33 views

Question about text classification without labeled data

I am working on a text classifier but at the moment I'm quite lost on what to do. The classes form a tree with three levels, for example, class A (level 1), class A.1 (level 2, subclass of A), and ...
0
votes
0answers
8 views

How to add new words into word embedding model?

Inspired by the this post, I am curious about how to add new words into trained existing word embedding without retraining the entire embedding? My guess is of the following: there is no such thing as ...
1
vote
0answers
38 views

Word embedding autoencoder

I'm trying to train a word embedding autoencoder, but it either doesn't train, or trains but doesn't make predictions. I know I'm doing something wrong, so any help is greatly appreciated. Here is my ...
3
votes
2answers
71 views

How can word2vec or BERT be used for previously unseen words

Is there any way to modify word2vec or BERT to extend finding out embeddings for words that were not in the training data? My data is extremely domain-specific and I don't really expect pre-trained ...
1
vote
1answer
30 views

For text classification, would a BoW or Word Embeddings based model ever be better than a Language Model?

I've done a bit of research, with this being the best as far as objectively measuring quality, but wanted to ask from a theoretical perspective if BoW-based models (e.g. using TF-IDF) or word ...
2
votes
1answer
18 views

Types of averages when analyzing sentences

I have a list of words and their frequencies in a text corpus. So there are words like "a", "what", "some" that have really high frequencies, and other like "...
0
votes
0answers
10 views

How to use AraVec word embeddings to build text classification model?

I am working with ArSarcasm dataset from Hugging Face. I have cleaned the tweets from all the noise with a few preprocessing steps, tokenized the tweets and lemmatized the tokens (snippet of code can ...
1
vote
0answers
33 views

Plotting cosine similarities in 3d space from word embeddings

I'm working on a project in which I want to estimate biases from a large corpus of newspaper articles using word2vec. Following this and this paper, biases are calculated by constructing dimension x ...
0
votes
0answers
25 views

Gensim fast text get vocab or word index

Trying to use gensim's fasttext, testing the sample code from gensim with a small change of replacing the arguement to ...
0
votes
1answer
354 views

Is it possible to add new vocabulary to BERT's tokenizer when fine-tuning?

I want to fine-tune BERT by training it on a domain dataset of my own. The domain is specific and includes many terms that probably weren't included in the original dataset BERT was trained on. I know ...
0
votes
1answer
19 views

Word Embeddings fastText in 50 dimension

Is there a fastText embedding in 50 dimensions? I'm aware of GloVe embedding is dimensions (50, 100, 200, 300) dimensions. I am trying to sentiment analysis with a very small dataset. If there is ...
2
votes
2answers
42 views

Why are words represented by frequency counts before embedding?

Before getting vector representations of words by embedding, the words are mapped to numbers. These numbers are chosen to be the frequency of that word in the dataset. Why does this convention exist? ...
0
votes
0answers
115 views

Can I average the BERT embeddings of multiple instances of the same word to get one vector representation of the word?

In the project I'm working on right now I would like to get one embedding for every unique lemma in a corpus. Could I get this by averaging the embeddings of every instance of a lemma? For example, ...
0
votes
2answers
74 views

Handling unknown words when making NER Models

I'm working on my custom Named Entity Recognition model that I'm making in Python's Keras lib. I have read that I should enumerate all words that are appearing, so that I get vectorized sequences. I ...
0
votes
0answers
10 views

TF-IDF to find technical terms

I have some sentences and I want to see whether or not they contain words that are technical terms. I was thinking of working with Wikipedia texts: finding the most common words in a certain article, ...
1
vote
1answer
114 views

Not clear about relative position bias

I've been reading the Swin Transformer paper and came across relative position bias concept. I'm not able to figure out how is it more effective than positional embeddings. I hope someone can explain ...
0
votes
0answers
10 views

Why accuracy didn't increase while loss reached nearly zero

I am trying to build a classifier using IMDB dataset. So I used a pre-trained Word2Vec model by google with a 300D vector for single words. here is the code: ...
0
votes
0answers
32 views

What are the advantages for working with LSTM with Word embedding and just LSTM

I am a beginner in Tensorflow. I am working on a dataset for sentiment analysis. I have used two kinds of methods one is LSTM and the other is LSTM with pre-trained word embeddings like (GloVe & ...
0
votes
1answer
18 views

applicability of relative similarity computation

I've computed the cosine similarity between a & b (=x) and ...
0
votes
1answer
21 views

What is the difference between sparse and dense corpra?

I didn't got the meaning or the difference between sparse and dense corpra here in this sentence "the reason is that Skip-gram works better over sparse corpora like Twitter and NIPS, while CBOW ...
0
votes
0answers
14 views

Weird entries in GloVe embeddings causing error

I'm trying to load GloVe embedding data, and when just printing out the words and their corresponding embeddings I get an anomaly. With the following code: ...
0
votes
1answer
21 views

Getting context-word pairs for a continuous bag of words model and other confusions

Suppose I have a corpus with documents: ...
0
votes
0answers
17 views

GloVe word embedding

I am new to word embeddings. I am working with a very small dataset of 1000 rows and 2 columns for sentiment analysis. Is there any way to create a subset GloVe embedding which only contains 20% of ...
0
votes
0answers
16 views

Understanding fastText

fastText is Facebook's open source software to obtain word embeddings (the original paper). Given a document indexed by $n$ and represented by list of n-gram ...
0
votes
1answer
23 views

Question regarding training data in word2vec - skip-gram

I have a very simple question regarding the training data in word2vec. In the skip-gram implementation, the training data (if I understand it correctly) is generated as pairs of words like it's shown ...
0
votes
1answer
10 views

Ensure trained word embeddings get high similarity with particular words

I am trying out my hand at training a Word2Vec model using gensim. I made a simple training file that basically had just one line repeated multiple times ...

1
2 3 4 5
9