Questions tagged [gensim]

gensim is the python library for topic modelling. multi-dimensional vector representation of words or sentences which preserves semantic meaning is computed through word2vec and doc2vec models.

Filter by
Sorted by
Tagged with
0 votes
0 answers
40 views

Are the word of women and men different when expressing their views on the same subject?

My data includes women's comments on X and Y and men's comments on X and Y. Each comment is of equal length. I will calculate how much different the word choice between men and women when commenting ...
user avatar
  • 9
0 votes
0 answers
7 views

Understanding output of gensim LDA topic modeling API

I was trying to understand gensim mallet wrapper for topic modeling as explained in this notebook. In point 11, it prepares corpus which if of format Term Document frequency: ...
user avatar
  • 137
1 vote
1 answer
16 views

Recommend products based on historical queries of other users

Given the user data as in the following: ...
user avatar
0 votes
0 answers
6 views

gensim word2vec results - why non-nearby word first?

from gensim.models import Word2Vec model = Word2Vec(sentences = [['a','b'],['c','d']], window = 9999999, min_count=1) model.wv.most_similar('a', topn=10) Above ...
user avatar
0 votes
0 answers
18 views

How to find similar document (using gensim) given two or more other documents?

I am developing a similarity program to compare documents, and I’ve successfully trained my model with Gensim (TFIDF and LSI) in order to compare two documents of each other, and it works great. I can ...
user avatar
  • 101
-1 votes
2 answers
258 views

How to fit Word2Vec on test data?

I am working on a Sentiment Analysis problem. I am using Gensim's Word2Vec to vectorize my data in the following way: ...
user avatar
  • 1,241
0 votes
1 answer
233 views

How to calculate the mean average of word embedding and then compare strings using sklearn.metrics.pairwise

I am totally new to this topic, that's why I am so confused or stuck in this code for a while, but I am not sure how to solve it correctly. My goal is to write a short text embedding using vector ...
user avatar
  • 1
0 votes
0 answers
38 views

Is gensim.models.word2vec pretrained?

If you load the gensim word2vec model like this gensim.models import word2vec model = Word2Vec(my_corpus) is it pre-trained on some data already (Other than the ...
user avatar
2 votes
2 answers
123 views

How to examine if a Doc2Vec model is sufficiently trained?

I started experimenting with gensim's Doc2Vec for sentiment analysis. For the training of the embedding itself, I have seen examples using a reduced learning rate with a few 10s or even a few hundred ...
user avatar
  • 121
0 votes
0 answers
30 views

Training fasttext on your own corpus

I want to train fasttext on my own corpus. However, I have a small question before continuing. Do I need each sentences as a different item in corpus or can I have many sentences as one item? For ...
user avatar
  • 103
1 vote
1 answer
49 views

Why do we calculate the vector of a document by averaging the vectors of all the words?

I am trying to build a search engine to query a folder of documents. Tutorials online suggest that we should obtain the vector of a document by averaging the vectors of all the words, then compare ...
user avatar
  • 51
1 vote
0 answers
20 views

default estimation method of gensim's word2vec skipgram?

I am now trying to use word2vec by estimating skipgram embeddings via NCE (noise contrastive estimation) rather than conventional negative sampling method, as a recent paper did (https://asistdl....
user avatar
  • 11
0 votes
0 answers
15 views

How to handle words not in the dictionary (while finding similar words)?

I am doing a project on Semantic text analysis where my data has column Technical skills (so I have to train data to find similar words) which are words and not sentences. So I wish to find similar ...
user avatar
  • 33
0 votes
0 answers
51 views

Gensim fast text get vocab or word index

Trying to use gensim's fasttext, testing the sample code from gensim with a small change of replacing the arguement to ...
user avatar
0 votes
0 answers
16 views

How to find similar Sentences using FastText ( Sentences with Out of Vocabulary words)

I am trying to create an NLP model which can find similar sentences. For example, It should be able to say that "Software Engineer", "Software Developer", "Software Dev", ...
user avatar
4 votes
1 answer
432 views

Does spaCy support multiple GPUs?

I was wondering if spaCy supports multi-GPU via mpi4py? I am currently using spaCy's nlp.pipe for Named Entity Recognition on a high-performance-computing cluster ...
user avatar
0 votes
0 answers
39 views

Genesis most_similar find synonym only (not antonyms)

Is there a way to let model.wv.most_similar in gensim return positive-meaning words only (i.e. that shows synonyms but not antonyms)? For example, if I do: ...
user avatar
2 votes
1 answer
612 views

Fine-tuning pre-trained Word2Vec model with Gensim 4.0

With Gensim < 4.0, we can retrain a word2vec model using the following code: ...
user avatar
  • 51
0 votes
0 answers
28 views

LSA Model Improvement

I followed gensim's Core Tutorial and build an LSA Classification, topic modeling and Document Similarity model for newsgroups dataset. My code is available here. I need help with below 3 concepts. ...
user avatar
  • 1
2 votes
2 answers
137 views

classification of similar text input features with text output label

I hope somebody can provide guidance/input/advice on my project, where I believe AI can help. I have a general understanding of AI, but I lack a formal training. I've never built a neural net from ...
user avatar
  • 73
2 votes
0 answers
22 views

Evaluate Topic Modelling on synthetic data

I try to find the optimal number of topics on a synthetic corpus (so a list of lists of tokens I generate using various parameters). I, therefore, know the true number of topics and the true topics ...
user avatar
5 votes
1 answer
606 views

Why is averaging the vectors required in word2vec?

While implementing word2vec using gensim by following few tutorials online, one thing that I couldn't understand is the reason why word vectors are averaged once the model is trained. Few example ...
user avatar
  • 163
2 votes
0 answers
151 views

Text preprocessing on corpus in pipeline before Gensim word2vec training

I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this: ...
user avatar
1 vote
1 answer
117 views

LDA topic model has 0-weight topics, is that normal?

While experimenting with different number of topics for the Gensim implementation of LDA, I found that for a high number of topics, the output often consists of topics with all weights equal to zero. ...
user avatar
  • 189
1 vote
1 answer
16 views

Watch list of Tweets with unknown model

I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True)...
user avatar
0 votes
1 answer
710 views

Get most likely topic per document in pandas dataframe using gensim

I am using gensim LDA to build a topic model for a bunch of documents that I have stored in a pandas data frame. Once the model is built, I can call ...
user avatar
  • 167
1 vote
0 answers
45 views

doc2vec - paragraph or article as document

I'm trying to train a doc2vec model on the German wiki corpus. While looking for the best practice I've found different possibilities on how to create the training data. Should I split every Wikipedia ...
user avatar
  • 143
2 votes
1 answer
321 views

How to test the quality of a word embedding?

I have trained a word2vec model using GenSim 4. The problem is that my corpus is quite small. How can I test the quality of the word embeddings I have obtained? Is there some standard measures to do ...
user avatar
0 votes
1 answer
139 views

Difference between Word Embedding and Text Embedding

I am working on a dataset of amazon alexa reviews and wish to cluster them in positive and negative clusters. I am using Word2Vec for vectorization so wanted to know the difference between Text ...
user avatar
5 votes
1 answer
921 views

How to choose threshold for gensim Phrases when generating bigrams?

I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA ...
user avatar
  • 151
0 votes
2 answers
623 views

Error in using sklearn's GridSearchCV on Word2Vec

I am using the sklearn_api of gensim to create an estimator for a Word2vec model to pass it to sklearn's gridsearch . My code is as follows : ...
user avatar
  • 157
0 votes
1 answer
105 views

Convert bin model to pickle [closed]

I trained a word2vec model using Gensim library which is of type .bin Q1: can we convert this trained model in bin format to pickle? Q2: would it speed up the execution time?
user avatar
1 vote
1 answer
112 views

Two questions about word2vec and gensim

I've written the code below to try word2vec implementation of gensim. I've two questions: Even though I've removed stop words, the word "the" is listed as one of the most similar words of &...
user avatar
  • 113
2 votes
3 answers
101 views

How to identify text similarity based on training data?

I have a set of documents (1 to 11) for which the labeling is done. Lets Assume: ...
user avatar
3 votes
1 answer
118 views

How to work with different Encoding for Foreign Languages

I've got a Word Embedding File called model.txt. This contains 100 Dimensional vectors for over a million French words. These words contain accented characters such ...
user avatar
1 vote
0 answers
337 views

Extracting vectors of FastText own model to use it on a NN

I have trained my own model of fasttext using the pretrained model of English available on their website with the next code: ...
user avatar
  • 111
3 votes
1 answer
1k views

Predicting the missing word using fasttext pretrained word embedding models (CBOW vs skipgram)

I am trying to implement a simple word prediction algorithm for filling a gap in a sentence by choosing from several options: Driving a ---- is not fun in London streets. Apple Car Book King With ...
user avatar
2 votes
1 answer
27 views

Topic modelling on only 24 documents gives the same "topic" for any K

Description: I have 24 documents, each one of around 2.5K tokens. They are public speeches. My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
user avatar
  • 31
0 votes
1 answer
324 views

What's wrong with RF/SVM with word embedding (GloVe)?

I searched many times in google for examples on word embedding (specifically GloVe) with Random forest and I couldn't find any single example. For GloVe, it was all either LSTM or CNN. Maybe there's ...
user avatar
2 votes
1 answer
76 views

Annotating the vocabulary using Word2vec model

I am trying to label the vocabulary in the corpus. I have trained the word2vec model on the corpus I have grouped the words which are related based on the score as key as the first word as the key ...
user avatar
  • 51
0 votes
0 answers
26 views

To map topic to a document after topic modeling is done with LDA

Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and ...
user avatar
  • 11
0 votes
1 answer
123 views

Siamese networks vs Semantic similarity (may be gensim)

I am trying to understand the Siamese networks . In this vector is calculated for an object (say an image) and a distance metric is applied (say manhatten) on two vectors produced by the neural ...
user avatar
2 votes
1 answer
286 views

extract document topic vectors from lda model

how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation
user avatar
8 votes
1 answer
3k views

Difference between Gensim word2vec and keras Embedding layer

I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the ...
user avatar
  • 2,475
0 votes
1 answer
58 views

Models after word2vec outputs

I am originally using a bag of word (2-gram) model to approach a classification problem. The one hot encoding of the 2-gram output was sent to a logistic regression or neural network to build a ...
user avatar
  • 2,475
0 votes
1 answer
370 views

Length of document in doc2vec

I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired ...
user avatar
7 votes
0 answers
732 views

Gensim LDA model: return keywords based on relevance (λ - lambda) value

I am using the gensim library for topic modeling, more specifically LDA. I created my corpus, my dictionary, and my LDA model. With the help of the pyLDAvis library I visualized the results. When I ...
user avatar
1 vote
1 answer
85 views

how to do topic modeling on very huge data?

When i come to know that gensim is useful library for topic modeling, I tried it on my huge amount of document. It works well only if the dictionary size is to be fix. In my case i have each and every ...
user avatar
3 votes
1 answer
657 views

Metrics for unsupervised doc2vec model

I have just built a simple doc2vec model using the gensim library, pretty much followed the tutorial located here. The methods provided for checking the quality of the model are very manual and ...
user avatar
  • 161
1 vote
0 answers
964 views

How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?

I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data. At present I ...
user avatar
  • 11