Questions tagged [word-embeddings]
Word embedding is the collective name for a set of language modeling and feature learning techniques in NLP where words are mapped to vectors of real numbers in a low dimensional space, relative to the vocabulary size.
507
questions
0
votes
0
answers
35
views
Multimodal semantic search
Context: I am interested in using the potential of using embeddings to consolidate texts/documents with very different surface forms into one searchable database - in other words, to produce a sort of ...
- 121
0
votes
0
answers
26
views
Understanding gradient of skip gram
I am trying to understand gradient calculation for skip gram with softmax output and cross entropy loss.
I am referring these articles: 1, 2, 3.
The all calculate the error as follows:
$$E=-\sum_{c=1}...
- 103
3
votes
1
answer
33
views
Is there a reference dataset for contextual similarity?
I'm doing some experiments with word embeddings to try to capture context-aware similarity, so that for example the word pair apple - hardware, are very dissimilar in the context of a fruit store, but ...
- 241
0
votes
0
answers
9
views
Optimize wordembedding and neural network at the same time
I have a lot of (domain)-specific text that I want to classify into 100+ categories.
I want to train a wordembedding (FastText) and use that in conjuction with a CNN, thus I'm running into the problem ...
- 460
0
votes
0
answers
346
views
Can BM25 be used as an embedding algorithm?
I'v studied about BM25 algorithm. Untill now, I couldn't find an implementation of BM25 to give me an embedding of a text like TfidfTransformer and ...
0
votes
0
answers
178
views
Finetuning fasttext with unlabeled text corpus
I am training a classifier which is supposed to take the name of a product as input. For this purpose I want to finetune a pre-existing fasttext model on my article names.
My code looks like this
<...
1
vote
1
answer
52
views
Understand the interpretability of word embeddings
When reading the Tensorflow tutorial for Word Embeddings, I found two notes that confuse me:
Note: Experimentally, you may be able to produce more interpretable embeddings by using a simpler model. ...
- 155
2
votes
1
answer
1k
views
How do we evaluate the outputs of text generation models?
Evaluation of a wide variety of natural language generation (NLG) tasks is difficult. For instance, for a question answering model, it is hard for a human to quantify how well the model has answered a ...
- 121
0
votes
1
answer
317
views
Word2vec CBOW model with negative sampling
From this article:
In vanilla skip gram model, softmax is computationally very expensive, as
it requires scanning through the entire output embedding matrix (W_output) to compute the probability ...
- 179
0
votes
0
answers
12
views
Calculating noise distribution in skip gram negative sampling
I was referring to this article explaining skip-gram with negative sampling. It says we need to sample negative samples from noise distribution calculated as follows:
$$P_n(w) = \left(\frac{U(w)}{Z}\...
- 179
0
votes
0
answers
19
views
Does word2vec skip gram involves softmax in the output layer
I was going through various pytorch and from-scratch implementations of skip-gram. I found following:
This implementaiton does not seem to use softmax
...
- 103
1
vote
1
answer
93
views
Binary Classification [Text] based on Embedding Distance?
I was just informed this community was a better fit for my SO question. I am wondering if I can use a Milvus or Faiss (L2 or IP or...) to classify documents as similar or not based on distance. I have ...
0
votes
1
answer
55
views
Change the shape of numpy array
My numpy array has the shape of (99,2) basically it has 2 columns one is the word and the other is a hot encoding vector size of 300 I want to delete the column of words and want to have only encoding ...
2
votes
1
answer
6k
views
Which word embedding mechanism does chatGPT use?
Which word embedding mechanism does chatGPT use? Is it Word2Vec, GloVe, or something else?
- 123
0
votes
0
answers
25
views
Neural network architecture for classification on Word2Vec embeddings
tl; dr
I'm training a supervised model using 100-dimensional Word2Vec embeddings as features. What kind of neural network architecture is optimal for this use case?
A full explanation of what I'm ...
- 133
0
votes
0
answers
24
views
Select Topic Words from Clusters
I am following this solution for clustering: https://towardsdatascience.com/clustering-contextual-embeddings-for-topic-model-1fb15c45b1bd
For step four "Select Topic Words from Clusters", I ...
- 129
0
votes
1
answer
62
views
Get the value of second dimension in numpy array
My NumPy array looks like this
...
0
votes
2
answers
92
views
Similarity with respect to a specific concept in text embeddings
In text embeddings, cosine similarity is often used to find texts similar to a search query.
However, I don't want to find a text that is overall similar, but similar with regards to a specific ...
- 101
0
votes
0
answers
11
views
How to storge large data values in data frame or what would be suitable way?
My output has two columns, basically one is a token name and the second column is the embedding which is the array consisting of 300 values, if I store these tables in CSV files, most of the array ...
0
votes
1
answer
29
views
How word2vect algorithm works using a neural network
Can anyone provide information as to how a word2vec algorithm works using a neural network.
(An easy example to understand it with formulas please.)
- 3
0
votes
1
answer
287
views
Train Word Embeddings on new vocabulary given the pre trained embeddings through word2vec
I have the pre-trained Embbedings on the language. I have the vocabulary for that language, what would be the pipeline to train this vocabulary by using Pre train embeddings through the word2vec model?...
0
votes
1
answer
477
views
How is weight matrix calculated in a neural network?
Context: I am a pure mathematician trying to understand machine learning. I am studying it from various sources, now focusing on NLP and word embeddings.
My question: What is the weight matrix for a ...
- 103
0
votes
0
answers
68
views
How assign a new sentence embedding to a cluster in fast clustering
I am working on clustering sentence embeddings obtained from sentence transformer, for which I used fast clustering (clustered on cosine similarity).
https://www.sbert.net/examples/applications/...
- 23
3
votes
2
answers
599
views
Gensim doc2vec error: KeyError: "word 'senseless' not in vocabulary"
I am new to machine learning and tried doc2vec on quora duplicate dataset. new_dfx has columns 'question1' and 'question2' which has preprocessed questions in each row. Following is the tagged ...
1
vote
1
answer
37
views
Which pre-trained model to select to generate embeddings from shop names written in English?
Good afternoon!
I have a dataset with thousands of shop names written in English. Several shop names might belong to one business entity, for instance, shops with names "KFC 001", "WWW....
- 11
1
vote
1
answer
137
views
Using the whole GloVe pre-trained embedding matrix or minimize the matrix based on the number of words in vocabulary
I have created a neural network for sentiment analysis using bidirectional LSTM layers and pre-trained GloVe embeddings.
During the training I noticed that the ...
- 111
0
votes
1
answer
147
views
What does Embeddings Array Represent in BERT's Feature Extraction?
I am new to academic NLP, and I had been tasked with to use BERT to extract features of a sentence.
...
- 23
1
vote
0
answers
39
views
Is normalization of word embeddings important?
I am doing actor-critic reinforcement learning for an environment that is best represented as a "bag-of-words". For this reason, I have opted to use a single body, multi-head approach for ...
1
vote
1
answer
320
views
NLP Question: Where can I find a list of all open compound words (with spaces in them), like "peanut butter" or "high school"? [close
I already have a list of "1-gram" words, which include closed compound words like "skyscraper" or "weatherman."
However, I'm also interested in compiling a list of "...
- 13
2
votes
4
answers
515
views
Algorithm to determine whether the first row in CSV is likely to be a header row or a data row
I have a fairly simple problem. I am trying to determine whether the first row in CSV is likely to be a header row or a data row. Looking at single column, the problem can be simplified to: I have a ...
- 121
1
vote
2
answers
190
views
How to interpret differences between 2D and 3D T-SNE visualization of similar words from Word2Vec embedding?
I have created a Word2Vec model based on the transcript of the Office. I am now trying to visualize the embedding space for the top similar words of an input word with t-SNE in 2D and 3D. I ...
- 145
1
vote
1
answer
97
views
How to vectorize newline \n in tensorflow textVectorization layer?
I am working on text generation model and i want to vectorize the newline character '\n' as a word in tensorflow. How DO i do it.
I have done this so far. but tensorflow just not consider it.
...
- 135
1
vote
0
answers
67
views
Machine learning with mixed variables dataset (numerical, categorical and embeddings)
I'm working on a machine learning project where I'm trying to predict the revenue of a movie.
My dataset contains mixed data types. There are numerical features (rating, number of votes, release year,....
- 111
0
votes
0
answers
80
views
Word embedding for Non-NLP words
I would like to embed words with a context, but that is not "Natural Language" - but just a list of words about more or less the same topic. Is there a way to use this context for the ...
- 323
0
votes
2
answers
538
views
Alternatives to word to vector embedding
I'm just curious are there some alternative techniques to word 2 vector representation? So words/phrases/sentences are not represented as vectors but have a different form. Thanks.
- 99
1
vote
1
answer
22
views
Does word ordering affect monolingual alignment success
I spent some time reading about both Word2Vec embeddings and alignment between different embeddings (for instance vecmap) and was wondering whether there is any significance to the word ordering of ...
- 111
2
votes
1
answer
183
views
Does Word2Vec's skip-gram NNLM even produce context words?
Let me first establish what CBoW and skip-gram are supposed to do. You can skip to the next section if you think this is unnecessary.
Background
My understanding is that Word2Vec is a suite of 2 ...
- 193
1
vote
1
answer
438
views
How to fine-tune hyperameters of unsupervised training in fasttext?
I want train fasttext unsupervised model on my text dataset. However there are many hyperparameters in train_unsupervised method:
...
- 183
0
votes
0
answers
11
views
Discover context in word alignement
I am using Facebook Muse to translate words from one language to another, and apparently it performs well (I set no metric though). Although I have very basic ML/Datascience/NLP knowledge, could you ...
- 453
0
votes
1
answer
222
views
Is it normal for a model to perform worse with the use of word embeddings?
I have a multiclass text classification problem and I've tried different solutions and models, but I was not satisfied with the results.
So I've decided to use GloVe ( Global Vectors for Word ...
- 13
0
votes
1
answer
108
views
Why GloVe model (by gensim) does not have vectors for numbers 1, 2, ...?
I expected GLoVe to have vectors for numbers.
from gensim import downloader as api
glove = api.load("glove-twitter-25")
glove['1']
This results in ...
- 101
2
votes
0
answers
118
views
how Can we add extra word embedding to the pytorch funnel transformer?
i was approaching NLP sequence classification problem (3 classes) using huggingface transformers (funnel-transformer/large) and tensorflow.
first i created laserembedding like this :
...
- 121
1
vote
2
answers
103
views
How does "A Neural Probabilistic Language Model" learn good word vectors?
I'm a layman making a foray into NLP and I have a question: The landmark paper A Neural Probabilistic Language Model (Bengio et al., 2003) makes an attempt at statistical language modelling by (1) ...
- 11
0
votes
1
answer
93
views
Clustering unknown product names
I have a parser that reads messages that contain product names. I would like to automatically cluster product names in clusters where each cluster would be one product and all the ways it can be ...
- 143
0
votes
1
answer
80
views
weights of coocurrence matrix in glove
I was studying the theory behind glove and was checking out some implementations of it. Before passing the data to its neural networks, I noticed that the weights of the co-occurrence matrix aren't ...
- 111
1
vote
1
answer
155
views
Why do RNN text generation models treat word prediction as a classification task?
In many of the sources I have found regarding text generation with word-based RNN models (LSTM or GRU), the model is trained to perform a classification task across the vocabulary (such as with ...
- 113
2
votes
1
answer
395
views
Range of values of BERT and other embeddings?
Are the values in all NLP models' embeddings between the range -1 to 1?
If not, what models use a different range (or decimal points)? And what could be the reason for that shift/change?
- 133
0
votes
0
answers
309
views
Using BERT embeddings as input for transformer architecture
I will use BERT's embedding weights (as discussed here) for embedding in embedding layers of the transformer model. But my question is: don't embeddings of BERT already go through the whole encoding ...
- 91
1
vote
0
answers
407
views
What are the inputs of encoder and decoder layers of transformer architecture?
In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the ...
- 91
1
vote
0
answers
41
views
Which model is better able to understand the difference that two sentences are talking about different things?
I'm currently working on the task of measuring semantic proximity between sentences. I use fasttext train _unsiupervised (skipgram) for this. I extract the sentence embeddings and then measure the ...
- 183