Questions tagged [gensim]

gensim is the python library for topic modelling. multi-dimensional vector representation of words or sentences which preserves semantic meaning is computed through word2vec and doc2vec models.

Filter by
Sorted by
Tagged with
4
votes
1answer
220 views

Why is averaging the vectors required in word2vec?

While implementing word2vec using gensim by following few tutorials online, one thing that I couldn't understand is the reason why word vectors are averaged once the model is trained. Few example ...
1
vote
0answers
32 views

Text preprocessing on corpus in pipeline before Gensim word2vec training

I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this: ...
1
vote
1answer
17 views

LDA topic model has 0-weight topics, is that normal?

While experimenting with different number of topics for the Gensim implementation of LDA, I found that for a high number of topics, the output often consists of topics with all weights equal to zero. ...
0
votes
0answers
13 views

Best way to build gensim WdmSimilarity for document data

I'm building an application that searches for queries in OCR data. My documents are numerous and have many pages. I'm using Wmd Similarity to query my data with gensim ...
1
vote
0answers
14 views

Watch list of Tweets with unknown model

I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True)...
0
votes
1answer
68 views

Get most likely topic per document in pandas dataframe using gensim

I am using gensim LDA to build a topic model for a bunch of documents that I have stored in a pandas data frame. Once the model is built, I can call ...
1
vote
0answers
15 views

doc2vec - paragraph or article as document

I'm trying to train a doc2vec model on the German wiki corpus. While looking for the best practice I've found different possibilities on how to create the training data. Should I split every Wikipedia ...
2
votes
1answer
40 views

How to test the quality of a word embedding?

I have trained a word2vec model using GenSim 4. The problem is that my corpus is quite small. How can I test the quality of the word embeddings I have obtained? Is there some standard measures to do ...
0
votes
1answer
31 views

Difference between Word Embedding and Text Embedding

I am working on a dataset of amazon alexa reviews and wish to cluster them in positive and negative clusters. I am using Word2Vec for vectorization so wanted to know the difference between Text ...
3
votes
0answers
381 views

How to choose threshold for gensim Phrases when generating bigrams?

I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA ...
0
votes
2answers
280 views

Error in using sklearn's GridSearchCV on Word2Vec

I am using the sklearn_api of gensim to create an estimator for a Word2vec model to pass it to sklearn's gridsearch . My code is as follows : ...
0
votes
1answer
38 views

Convert bin model to pickle [closed]

I trained a word2vec model using Gensim library which is of type .bin Q1: can we convert this trained model in bin format to pickle? Q2: would it speed up the execution time?
0
votes
0answers
34 views

How to compare cosine distances across two groups of words?

I am using a word2vec model based on Wikipedia corpus. I was looking for a way to quantify if two sets of words - s1= {a1, a2...}...
1
vote
1answer
49 views

Two questions about word2vec and gensim

I've written the code below to try word2vec implementation of gensim. I've two questions: Even though I've removed stop words, the word "the" is listed as one of the most similar words of &...
2
votes
2answers
36 views

How to identify text similarity based on training data?

I have a set of documents (1 to 11) for which the labeling is done. Lets Assume: ...
3
votes
1answer
72 views

How to work with different Encoding for Foreign Languages

I've got a Word Embedding File called model.txt. This contains 100 Dimensional vectors for over a million French words. These words contain accented characters such ...
1
vote
0answers
101 views

Extracting vectors of FastText own model to use it on a NN

I have trained my own model of fasttext using the pretrained model of English available on their website with the next code: ...
2
votes
1answer
564 views

Predicting the missing word using fasttext pretrained word embedding models (CBOW vs skipgram)

I am trying to implement a simple word prediction algorithm for filling a gap in a sentence by choosing from several options: Driving a ---- is not fun in London streets. Apple Car Book King With ...
2
votes
0answers
18 views

Topic modelling on only 24 documents gives the same “topic” for any K

Description: I have 24 documents, each one of around 2.5K tokens. They are public speeches. My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
0
votes
1answer
183 views

What's wrong with RF/SVM with word embedding (GloVe)?

I searched many times in google for examples on word embedding (specifically GloVe) with Random forest and I couldn't find any single example. For GloVe, it was all either LSTM or CNN. Maybe there's ...
2
votes
1answer
62 views

Annotating the vocabulary using Word2vec model

I am trying to label the vocabulary in the corpus. I have trained the word2vec model on the corpus I have grouped the words which are related based on the score as key as the first word as the key ...
0
votes
0answers
18 views

To map topic to a document after topic modeling is done with LDA

Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and ...
0
votes
1answer
69 views

Siamese networks vs Semantic similarity (may be gensim)

I am trying to understand the Siamese networks . In this vector is calculated for an object (say an image) and a distance metric is applied (say manhatten) on two vectors produced by the neural ...
1
vote
1answer
134 views

extract document topic vectors from lda model

how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation
7
votes
1answer
2k views

Difference between Gensim word2vec and keras Embedding layer

I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the ...
0
votes
1answer
42 views

Models after word2vec outputs

I am originally using a bag of word (2-gram) model to approach a classification problem. The one hot encoding of the 2-gram output was sent to a logistic regression or neural network to build a ...
0
votes
1answer
197 views

Length of document in doc2vec

I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired ...
5
votes
0answers
510 views

Gensim LDA model: return keywords based on relevance (λ - lambda) value

I am using gensim library for topic modeling, more specifically LDA. I have created my corpus, my dictionary and my lda model, and with the help of pyLDAvis library I visualize the results. When I ...
1
vote
1answer
61 views

how to do topic modeling on very huge data?

When i come to know that gensim is useful library for topic modeling, I tried it on my huge amount of document. It works well only if the dictionary size is to be fix. In my case i have each and every ...
2
votes
1answer
492 views

Metrics for unsupervised doc2vec model

I have just built a simple doc2vec model using the gensim library, pretty much followed the tutorial located here. The methods provided for checking the quality of the model are very manual and ...
1
vote
0answers
792 views

How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?

I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data. At present I ...
0
votes
1answer
759 views

Does gensim use Negative sampling in Word2vec?

When I train a word2vec model in Gensim on a huge amount of words/data (let's say hundreds of thousands of word vectors), is gensim using negative sampling automatically? Alternatively, is there a ...
1
vote
0answers
88 views

Copying embeddings for gensim word2vec

I wanted to see if I can simply set new weights for gensim's Word2Vec without training. I get the 20 News Group data set from scikit-learn (from sklearn.datasets import fetch_20newsgroups) and trained ...
0
votes
0answers
54 views

Linking LDA topics to the input documents

I am new to LDA topic modelling. I am using gensim and am able to generate topics that make sense. Using 25k of documents, I can also print them using print_topics. ...
4
votes
1answer
770 views

word2vec word embeddings creates very distant vectors, closest cosine similarity is still very far, only 0.7

I started using gensim's FastText to create word embeddings on a large corpus of a specialized domain (after finding that existing open source embeddings are not performing well on this domain), ...
1
vote
0answers
45 views

How to create pretrained word embedding text file with additional word features

I've had an idea for using word features to improve the quality of neural machine translation. Now, I would like to create word embeddings with additional word features such as pos tag, named entity, ...
10
votes
3answers
16k views

Word2Vec how to choose the embedding size parameter

I'm running word2vec over collection of documents. I understand that the size of the model is the number of dimensions of the vector space that the word is embedded into. And that different dimensions ...
3
votes
4answers
5k views

How to train an existing word2vec gensim model on new words?

According to gensim docs, you can take an existing word2vec model and further train it on new words. The training is streamed, meaning sentences can be a generator, reading input data from disk ...
2
votes
1answer
2k views

Sentence similarity using Doc2vec

I have a list of 50k sentences such as : 'bone is making noise', 'nose is leaking' ,'eyelid is down' etc.. I'm trying to use Doc2Vec to find the most similar sentence from the 50k given a new ...
1
vote
1answer
221 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
1
vote
0answers
95 views

Can I use Gensim doc2vec model for classification new documents?

Can I use Gensim doc2vec model for classification new documents via infer_vector? All my tests gave too bad results, even for big datasets (10GB utf-8 texts)...
1
vote
1answer
928 views

Fasttext error while loading wiki pre-trained data

I am loading the model using gensim package this way: from gensim.models import FastText model = FastText.load_fasttext_format('wiki-news-300d-1M-subword.bin') ...
1
vote
0answers
75 views

Doc2Vec for dataset with several text fields: concatenate or separate models?

I have a dataset with several fields: description, name, header. I want to train doc2vec out of it, so that I could use vectors for classification. So I wonder, ...
0
votes
1answer
94 views

Understanding word2vec vectors representation

I'm trying to obtain the word2vec representation of few words using gensim. At present, this is the model that I have: ...
2
votes
1answer
710 views

Updating Google News Word2vec Word Embedding?

Is it possible to update the Google News Word Embedding with a custom text dataset (text data pertaining to a particular domain) ? Google News Word2Vec - Word Embedding clearly helps us to come with ...
1
vote
0answers
137 views

Can we use doc2vec to detect outlier documents?

I have a set of documents and I want to identify and remove the outlier documents. I am just wondering if doc2vec can be used for this task. Or are there any recently evolved, promising algorithms ...
3
votes
2answers
2k views

can I use public pretrained word2vec, and continue train it for domain specific text?

I have a set of reviews from apparel domain, about 100K reviews (2M words). And I want to train word2vec to do some cool NLP staff with it. However the size is not enough for creating adequate ...
2
votes
1answer
844 views

How to count number of word embeddings in Gensim Word2Vec model

I am trying to create a Word2Vec model of the the Pub Med Central corpus using the Gensim library and want to limit the total number of word embeddings to around 1 billion. I have searched high and ...
1
vote
1answer
1k views

how to update the pre-trained word2vec model with new train data using genism

Hi I have used the genism to load the Spanish fasttext word2vec model with following code: ...
0
votes
1answer
245 views

what does this doc2vec based ML predict?

I'm trying to understand what does this ML program - which based on doc2vec - predict: ...