Questions tagged [word2vec]

word2vec is a two layer neural network to process text. It takes words as an input and outputs a vector correspondingly. It uses a combination of Continuous Bag of Word and skipgram model implementation.

Filter by
Sorted by
Tagged with
2
votes
2answers
22 views

How can word2vec or BERT be used for previously unseen words

Is there any way to modify word2vec or BERT to extend finding out embeddings for words that were not in the training data? My data is extremely domain-specific and I don't really expect pre-trained ...
0
votes
0answers
7 views

How to use AraVec word embeddings to build text classification model?

I am working with ArSarcasm dataset from Hugging Face. I have cleaned the tweets from all the noise with a few preprocessing steps, tokenized the tweets and lemmatized the tokens (snippet of code can ...
1
vote
0answers
26 views

Plotting cosine similarities in 3d space from word embeddings

I'm working on a project in which I want to estimate biases from a large corpus of newspaper articles using word2vec. Following this and this paper, biases are calculated by constructing dimension x ...
0
votes
0answers
6 views

Choosing an explainable embedding and classifier when each document only have one sentence

I have dataset with corpus of 20K documents. Each document is a short 1 sentences. I need to classify each sentence in 0/1 classes as well as being able to point exactly what words are responsible for ...
0
votes
0answers
13 views

Attention to get context of words

The W2V techniques define context as a window of k words around the term, and using this learn the vector representations for words in the corpus. Attention networks can help us get the important ...
0
votes
1answer
11 views

Data Set and guidance for Occupations/ Roles classification problem

I am working on a project where I need to find similar roles -- for example, Software Engineer, Soft. Engineer , Software Eng ( all should be marked similar) Currently, I have tried using the Standard ...
0
votes
0answers
10 views

Why accuracy didn't increase while loss reached nearly zero

I am trying to build a classifier using IMDB dataset. So I used a pre-trained Word2Vec model by google with a 300D vector for single words. here is the code: ...
0
votes
1answer
19 views

What is the difference between sparse and dense corpra?

I didn't got the meaning or the difference between sparse and dense corpra here in this sentence "the reason is that Skip-gram works better over sparse corpora like Twitter and NIPS, while CBOW ...
0
votes
0answers
8 views

Weird entries in GloVe embeddings causing error

I'm trying to load GloVe embedding data, and when just printing out the words and their corresponding embeddings I get an anomaly. With the following code: ...
0
votes
1answer
18 views
0
votes
0answers
11 views

Genesis most_similar find synonym only (not antonyms)

Is there a way to let model.wv.most_similar in gensim return positive-meaning words only (i.e. that shows synonyms but not antonyms)? For example, if I do: ...
0
votes
1answer
18 views

Question regarding training data in word2vec - skip-gram

I have a very simple question regarding the training data in word2vec. In the skip-gram implementation, the training data (if I understand it correctly) is generated as pairs of words like it's shown ...
1
vote
0answers
112 views

Fine-tuning pre-trained Word2Vec model with Gensim 4.0

With Gensim < 4.0, we can retrain a word2vec model using the following code: ...
0
votes
1answer
25 views

Kmeans with Word2Vec model unexpected results

I'm trying to play around with unsupervised NLP using Word2Vec. So far, the data i used is very small, but that is because I am just testing to see how Kmeans will work. The Kmeans was performed first ...
0
votes
1answer
31 views

How does Word2Vec actually help with sentimental analysis?

I'm trying read in a whole article, separate the article by sentences, and then words. Then I pass this into the Word2vec Model and the output comes out. However, my goal is to find the positive or ...
0
votes
1answer
15 views

How to use Word2Vec CBOW in statistical algorithm?

I have seen a few examples of using CBOW in neural network models (although I did not understand them). I know that Word2Vec is not similar to BOW or TFIDF, as there is no single value for CBOW and ...
0
votes
0answers
27 views

How to do trained glove/word2vec analogy test?

I've trained a glove and a word2vec model using a dataset with lots of news. My instructor told me to evaluate these models using this script. He told me to use the same dataset but I am getting the ...
0
votes
0answers
12 views

Checking whether the new vocabulary is not in the old vocabulary of word2vec using Gensim

How do you check whether the new vocabulary is not in the old vocabulary and then further train the existing model and see how the new vocabulary was incorporated in word2vec using gensim? Could ...
2
votes
1answer
55 views

Negative sampling for graph representation learning

I was watching a lecture about graph representation learning (from here) and got a little bit confusing about how they define the negative samping procedure. In the presentation J. Leskovec briefly ...
1
vote
0answers
35 views

Hyper parameters (window size and vector dimensions) tuning in word2vec using Grey Wolf Optimization

Using Grey wolf Optimization, I want to calculate optimal values of two hyper parameters: context window size and embedding size (vector dimensions) for word2vec skipgram model used for word embedding....
1
vote
0answers
89 views

How do I convert my dataframe column into vectors using word2vec?

I have the column of Categories of vary length in my DataFrame as shown below : Categories [Restaurants , Drinks] [Restaurants , Drinks , Rooftop] [Dinner] . . . ...
1
vote
1answer
22 views

Human readable format for clusters of word vectors

Let's say I have pretrained word2vec model and apply it to dataset consisting of article titles from "The Guardian". It seems pretty obvious that titles coming from "Science" ...
0
votes
1answer
29 views

Where the objective function of Skip-Gram Negative Sampling (SGNS) come from?

In the paper of "Neural Word Embedding as Implicit Matrix Factorization", there is an objective function for Skip-Gram Negative Sampling. I wanted to know where this formula come from
0
votes
1answer
42 views

Any research on relationship between the dimensions of a (word2Vec) space and how the human mind constructs meaning (or reality) through language?

Neuroscience is still trying to "find" how the mind (and language) somehow "works". Is there any theory linking a (low-dimensionality) embedding space (like word2Vec) to a mind (...
4
votes
1answer
360 views

Why is averaging the vectors required in word2vec?

While implementing word2vec using gensim by following few tutorials online, one thing that I couldn't understand is the reason why word vectors are averaged once the model is trained. Few example ...
2
votes
3answers
197 views

Word2vec outperforming BERT, possible?

I'm trying to solve a multilabel classification (dataset is tweet text) using a combination of BERT and CNN. As a benchmark, I'd compare it to other word embeddings, one of which is Word2vec. After ...
0
votes
1answer
34 views

Why does word2vec try to maximize the dot product of the center word vector and context?

I am learning about the maths behind word2vec from this tutorial. u are the embeddings for the center word and v for the context. It appears that this dot product is to be maximized. Why the context ...
2
votes
3answers
283 views

The reason behind using a pre-trained model?

These last month I have been studying all about word embeddings and the most known pre-trained word embeddings, Word2Vec, GloVe, FastText, etc. I have read many times how important It is to take ...
0
votes
1answer
28 views

Word2Vec: Identifying many-to-one relationships between words

Standard introductory examples in Word2Vec, like king - queen = man - woman and tokyo - japan = london - uk, involve one-to-one ...
0
votes
0answers
14 views

How to train effective domain specific word2vec models?

I'm training a word2vec model based on our custom data. Due to the size of data and the resources available to me, I'll use pyspark to train spark's built-in word2vec model. However, I'm curious to ...
2
votes
0answers
132 views

Difference between NCE-Loss and InfoNCE-Loss

I started looking into word2vec and was wondering what the connection/difference between the NCE-Loss and the infoNCE-Loss is. I get the basic idea of both. I have a hard time deriving one from ...
2
votes
0answers
96 views

Text preprocessing on corpus in pipeline before Gensim word2vec training

I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this: ...
1
vote
1answer
17 views

How can I convert my predictions to text after predicting using RNN?

I'm building PoS tagger for our language. I give tokens to the words and tags using Tokenizer(). Functions for word and tag are different. ...
0
votes
1answer
20 views

why does not CBOW of word2vec include center word in context

In CBOW, each word has input vector and output vector, or context vector and self vector, the context aggregate the surrounding words, what if the context also contain the input vector of the center ...
1
vote
0answers
26 views

What is the meaning of, or explanation for, having multiple tags in a Doc2Vec model's TaggedDocuments?

I've tried reading the other answers on this topic but I'm unsure if I understand completely. For my dataset, I have a series of tagged documents, "good" or "bad." Each document ...
0
votes
0answers
22 views

How to implement co-occurance matrix in Python which only takes into account next word relationships

I've been trying to create a dictionary version of a co-occurence matrix in Python, such that it only counts instances where one word follows another in a set of sentences. I found this implementation ...
0
votes
0answers
13 views

Skipgram model theory confusion

In the output layer of a skipgram model, there are $|\text{Context}|*|\text{Vocab}|$ values. And for each context word, the values are basically the dot product of the input word representation and ...
0
votes
0answers
10 views

Word2Vector multicontext CBOW model with Adam optimization

In cbow multiword context word2vec model there are two weights matrixes $$ W,W^{'}$$ Where $W$ is $I$ -> $H$ weight matrix, and $W^{'}$ is $H$ -> $U$ weight matrix and output is just softmax ...
0
votes
0answers
25 views

Clustering together words that appear together while down weighting words that appear too often

I was wondering if I could get some help finding a good model for the problem I have. I have a data set where each observation is a set of words that go together. So for example, it could be: ...
2
votes
1answer
38 views

Word2Vec vs. Doc2Vec Word Vectors

I am doing some analysis on document similarity and was also interested in word similarity. I know that doc2vec inherits from word2vec and by default trains using word vectors which we can access. My ...
0
votes
0answers
28 views

Return the most relevant columns for a given keyword

Suppose my database has column name and description for each column of each table. Need to design an interface where a user can enter a keyword and the interface will return the most relevance columns....
3
votes
3answers
462 views

Dot product for similarity in word to vector computation in NLP

In NLP while computing word to vector we try to maximize log(P(o|c)). Where P(o|c) is probability that o is outside word, given that c is center word. Uo is word vector for outside word Vc is word ...
2
votes
0answers
51 views

Is it acceptable to append information to word embeddings?

Let's say I have my 300 dimensional word embedding trained with Word2Vec and it contains 10,000 word vectors. I have additional data on the 10,000 words in the form of a vector (10,000x1), containing ...
0
votes
0answers
56 views

Adding extra feature to word2vec embeddings for use in classification

I have a word2vec model that will transform my text into vectors, and I need to use them to classify the text. My data is essentially a timeseries of chat messages, so I think that the message ...
4
votes
2answers
124 views

Word2Vec: Why do some dimensions of an embedding have an interpretation, and why does addition/subtraction of embedding vectors work?

I'm reading about Word2Vec from this source: http://jalammar.github.io/illustrated-word2vec/. Below is the heatmap of the embeddings for various words. In the source, it's claimed that we can get an ...
0
votes
0answers
22 views

Building simple documents search engine

I'm having my first steps in the NLP and at the moment I'm looking forward to building my own documents search engine. I've already got to know with TFIDF in practical way and I've also read about ...
0
votes
1answer
127 views

Continuous Bag Of Words (CBOW) network architecture?

Looking into word2vec like embeddings I found this exercise on PyTorch's website which prompts the reader to implement a CBOW network in PyTorch. My question is about the architecture to implement ...
0
votes
0answers
8 views

Projection layer function

I am trying to understand word-vectors and was reading this paper https://arxiv.org/pdf/1301.3781.pdf This paper proposes CBOW architecture which uses projection layer. What is projection layer? I ...
2
votes
1answer
127 views

How to test the quality of a word embedding?

I have trained a word2vec model using GenSim 4. The problem is that my corpus is quite small. How can I test the quality of the word embeddings I have obtained? Is there some standard measures to do ...
0
votes
0answers
59 views

Word2Vec vs LexVec vs GloVe

I'm on a NLP project and found a resource that has the three word representations mentioned in the question name, and I am struggling to find one place where they all are explained and compared. As I ...

1
2 3 4 5
7