Questions tagged [gensim]

gensim is the python library for topic modelling. multi-dimensional vector representation of words or sentences which preserves semantic meaning is computed through word2vec and doc2vec models.

Filter by
Sorted by
Tagged with
0 votes
1 answer
19 views

predict next career suggestion

I have a dataset having job and description. i want to make model which can predict what are the thing that user needs to improve when the user inputs his skills. For an example, If he has skills - ...
pycoder's user avatar
0 votes
0 answers
24 views

I want to make a Career suggestion model

There is a dataset having job titles and the descriptions. when a person enter his skills i need to output which category of job he should do. i have already created that using cosine similarity.(If ...
pycoder's user avatar
0 votes
0 answers
31 views

s3 object as corpus_file for gensim Doc2Vec

Is it possible to use a txt or jsonl file in an s3 bucket as the corpus_file input for a gensim Doc2Vec model? I am looking for something of the form: ...
Mutasim Mim's user avatar
0 votes
1 answer
168 views

Text segmentation problem

I am new to ML and trying to solve problem of text segmentation. I have a transcript of news show and I want to split this transcript into parts by topic. I tried to google and asked chatgpt and found ...
Oleg Bovykin's user avatar
0 votes
0 answers
21 views

Word2Vec Data Leak

I want to train a machine learning model that can determine the sentiment of tweets about different stocks. To do this I have a dataset, lets call it A. For dataset A about 30% of the data is labelled....
aaronm012's user avatar
3 votes
2 answers
737 views

Gensim doc2vec error: KeyError: "word 'senseless' not in vocabulary"

I am new to machine learning and tried doc2vec on quora duplicate dataset. new_dfx has columns 'question1' and 'question2' which has preprocessed questions in each row. Following is the tagged ...
Ankit Rohilla's user avatar
0 votes
1 answer
109 views

Gensim: create a dictionary from a large corpus without loading it in RAM?

The topic modelling library Gensim offers the ability to stream a large document instead of storing it in memory. Streaming is possible for the stage of converting the corpus to BOW, but the ...
Erwan's user avatar
  • 25.3k
1 vote
0 answers
332 views

How can I decide the threshold value for relevance score in a search problem?

I am using a LSA/TF-IDF/BM25/Ensemble models for text search and finally calculating similarity score to rank my search. I would like to decide a threshold value for the score, below which I would not ...
Prateek Coder's user avatar
0 votes
0 answers
44 views

Are the word of women and men different when expressing their views on the same subject?

My data includes women's comments on X and Y and men's comments on X and Y. Each comment is of equal length. I will calculate how much different the word choice between men and women when commenting ...
nem0's user avatar
  • 9
1 vote
1 answer
20 views

Recommend products based on historical queries of other users

Given the user data as in the following: ...
william007's user avatar
-1 votes
2 answers
2k views

How to fit Word2Vec on test data?

I am working on a Sentiment Analysis problem. I am using Gensim's Word2Vec to vectorize my data in the following way: ...
spectre's user avatar
  • 2,055
1 vote
2 answers
5k views

How to calculate the mean average of word embedding and then compare strings using sklearn.metrics.pairwise

I am totally new to this topic, that's why I am so confused or stuck in this code for a while, but I am not sure how to solve it correctly. My goal is to write a short text embedding using vector ...
test's user avatar
  • 11
2 votes
2 answers
672 views

How to examine if a Doc2Vec model is sufficiently trained?

I started experimenting with gensim's Doc2Vec for sentiment analysis. For the training of the embedding itself, I have seen examples using a reduced learning rate with a few 10s or even a few hundred ...
Shan Dou's user avatar
  • 131
0 votes
1 answer
232 views

Training fasttext on your own corpus

I want to train fasttext on my own corpus. However, I have a small question before continuing. Do I need each sentences as a different item in corpus or can I have many sentences as one item? For ...
BlueMango's user avatar
  • 113
1 vote
1 answer
253 views

Why do we calculate the vector of a document by averaging the vectors of all the words?

I am trying to build a search engine to query a folder of documents. Tutorials online suggest that we should obtain the vector of a document by averaging the vectors of all the words, then compare ...
amber's user avatar
  • 51
1 vote
0 answers
56 views

default estimation method of gensim's word2vec skipgram?

I am now trying to use word2vec by estimating skipgram embeddings via NCE (noise contrastive estimation) rather than conventional negative sampling method, as a recent paper did (https://asistdl....
krcoder's user avatar
  • 11
4 votes
2 answers
1k views

Does spaCy support multiple GPUs?

I was wondering if spaCy supports multi-GPU via mpi4py? I am currently using spaCy's nlp.pipe for Named Entity Recognition on a high-performance-computing cluster ...
Jinhua Wang's user avatar
2 votes
1 answer
3k views

Fine-tuning pre-trained Word2Vec model with Gensim 4.0

With Gensim < 4.0, we can retrain a word2vec model using the following code: ...
NST's user avatar
  • 51
0 votes
0 answers
35 views

LSA Model Improvement

I followed gensim's Core Tutorial and build an LSA Classification, topic modeling and Document Similarity model for newsgroups dataset. My code is available here. I need help with below 3 concepts. ...
Bala's user avatar
  • 1
2 votes
2 answers
304 views

classification of similar text input features with text output label

I hope somebody can provide guidance/input/advice on my project, where I believe AI can help. I have a general understanding of AI, but I lack a formal training. I've never built a neural net from ...
andrea's user avatar
  • 73
2 votes
0 answers
45 views

Evaluate Topic Modelling on synthetic data

I try to find the optimal number of topics on a synthetic corpus (so a list of lists of tokens I generate using various parameters). I, therefore, know the true number of topics and the true topics ...
seaslug95's user avatar
5 votes
2 answers
1k views

Why is averaging the vectors required in word2vec?

While implementing word2vec using gensim by following few tutorials online, one thing that I couldn't understand is the reason why word vectors are averaged once the model is trained. Few example ...
mockash's user avatar
  • 163
2 votes
0 answers
268 views

Text preprocessing on corpus in pipeline before Gensim word2vec training

I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this: ...
StraightUpBusta's user avatar
1 vote
1 answer
378 views

LDA topic model has 0-weight topics, is that normal?

While experimenting with different number of topics for the Gensim implementation of LDA, I found that for a high number of topics, the output often consists of topics with all weights equal to zero. ...
Emil's user avatar
  • 179
1 vote
1 answer
20 views

Watch list of Tweets with unknown model

I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True)...
cavalierstyles's user avatar
0 votes
1 answer
2k views

Get most likely topic per document in pandas dataframe using gensim

I am using gensim LDA to build a topic model for a bunch of documents that I have stored in a pandas data frame. Once the model is built, I can call ...
CopyOfA's user avatar
  • 167
1 vote
0 answers
62 views

doc2vec - paragraph or article as document

I'm trying to train a doc2vec model on the German wiki corpus. While looking for the best practice I've found different possibilities on how to create the training data. Should I split every Wikipedia ...
jonas's user avatar
  • 143
2 votes
1 answer
1k views

How to test the quality of a word embedding?

I have trained a word2vec model using GenSim 4. The problem is that my corpus is quite small. How can I test the quality of the word embeddings I have obtained? Is there some standard measures to do ...
robertspierre's user avatar
0 votes
1 answer
655 views

Difference between Word Embedding and Text Embedding

I am working on a dataset of amazon alexa reviews and wish to cluster them in positive and negative clusters. I am using Word2Vec for vectorization so wanted to know the difference between Text ...
Pari Ganjoo's user avatar
5 votes
1 answer
2k views

How to choose threshold for gensim Phrases when generating bigrams?

I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA ...
lefnire's user avatar
  • 151
0 votes
2 answers
1k views

Error in using sklearn's GridSearchCV on Word2Vec

I am using the sklearn_api of gensim to create an estimator for a Word2vec model to pass it to sklearn's gridsearch . My code is as follows : ...
Bharathi's user avatar
  • 277
0 votes
1 answer
331 views

Convert bin model to pickle [closed]

I trained a word2vec model using Gensim library which is of type .bin Q1: can we convert this trained model in bin format to pickle? Q2: would it speed up the execution time?
Vinay Sharma's user avatar
1 vote
1 answer
228 views

Two questions about word2vec and gensim

I've written the code below to try word2vec implementation of gensim. I've two questions: Even though I've removed stop words, the word "the" is listed as one of the most similar words of &...
Sanyo Mn's user avatar
  • 123
2 votes
3 answers
240 views

How to identify text similarity based on training data?

I have a set of documents (1 to 11) for which the labeling is done. Lets Assume: ...
Shivam Agrawal's user avatar
3 votes
1 answer
157 views

How to work with different Encoding for Foreign Languages

I've got a Word Embedding File called model.txt. This contains 100 Dimensional vectors for over a million French words. These words contain accented characters such ...
MetaInformation's user avatar
1 vote
0 answers
598 views

Extracting vectors of FastText own model to use it on a NN

I have trained my own model of fasttext using the pretrained model of English available on their website with the next code: ...
IMB's user avatar
  • 111
4 votes
1 answer
2k views

Predicting the missing word using fasttext pretrained word embedding models (CBOW vs skipgram)

I am trying to implement a simple word prediction algorithm for filling a gap in a sentence by choosing from several options: Driving a ---- is not fun in London streets. Apple Car Book King With ...
Kingstar's user avatar
2 votes
1 answer
43 views

Topic modelling on only 24 documents gives the same "topic" for any K

Description: I have 24 documents, each one of around 2.5K tokens. They are public speeches. My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
Luisda's user avatar
  • 31
0 votes
1 answer
541 views

What's wrong with RF/SVM with word embedding (GloVe)?

I searched many times in google for examples on word embedding (specifically GloVe) with Random forest and I couldn't find any single example. For GloVe, it was all either LSTM or CNN. Maybe there's ...
Dia Abujaber's user avatar
2 votes
1 answer
120 views

Annotating the vocabulary using Word2vec model

I am trying to label the vocabulary in the corpus. I have trained the word2vec model on the corpus I have grouped the words which are related based on the score as key as the first word as the key ...
Nuser's user avatar
  • 51
0 votes
0 answers
93 views

To map topic to a document after topic modeling is done with LDA

Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and ...
zirubak's user avatar
  • 11
1 vote
1 answer
245 views

Siamese networks vs Semantic similarity (may be gensim)

I am trying to understand the Siamese networks . In this vector is calculated for an object (say an image) and a distance metric is applied (say manhatten) on two vectors produced by the neural ...
Sandeep Bhutani's user avatar
2 votes
1 answer
799 views

extract document topic vectors from lda model

how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation
Syrinebh's user avatar
8 votes
1 answer
4k views

Difference between Gensim word2vec and keras Embedding layer

I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the ...
Edamame's user avatar
  • 2,745
0 votes
1 answer
90 views

Models after word2vec outputs

I am originally using a bag of word (2-gram) model to approach a classification problem. The one hot encoding of the 2-gram output was sent to a logistic regression or neural network to build a ...
Edamame's user avatar
  • 2,745
0 votes
1 answer
665 views

Length of document in doc2vec

I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired ...
Aishwarya A R's user avatar
8 votes
1 answer
1k views

Gensim LDA model: return keywords based on relevance (λ - lambda) value

I am using the gensim library for topic modeling, more specifically LDA. I created my corpus, my dictionary, and my LDA model. With the help of the pyLDAvis library I visualized the results. When I ...
Tasos Lytos's user avatar
1 vote
1 answer
114 views

how to do topic modeling on very huge data?

When i come to know that gensim is useful library for topic modeling, I tried it on my huge amount of document. It works well only if the dictionary size is to be fix. In my case i have each and every ...
Gaurav Koradiya's user avatar
3 votes
1 answer
840 views

Metrics for unsupervised doc2vec model

I have just built a simple doc2vec model using the gensim library, pretty much followed the tutorial located here. The methods provided for checking the quality of the model are very manual and ...
dendog's user avatar
  • 120
1 vote
0 answers
1k views

How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?

I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data. At present I ...
Vikrant's user avatar
  • 11