Questions tagged [gensim]
gensim is the python library for topic modelling. multi-dimensional vector representation of words or sentences which preserves semantic meaning is computed through word2vec and doc2vec models.
102
questions
0
votes
0
answers
25
views
s3 object as corpus_file for gensim Doc2Vec
Is it possible to use a txt or jsonl file in an s3 bucket as the corpus_file input for a gensim Doc2Vec model? I am looking for something of the form:
...
- 101
0
votes
1
answer
113
views
Text segmentation problem
I am new to ML and trying to solve problem of text segmentation.
I have a transcript of news show and I want to split this transcript into parts by topic. I tried to google and asked chatgpt and found ...
0
votes
0
answers
19
views
Word2Vec Data Leak
I want to train a machine learning model that can determine the sentiment of tweets about different stocks.
To do this I have a dataset, lets call it A. For dataset A about 30% of the data is labelled....
0
votes
0
answers
42
views
How to rank terms with bm25 and bo1 pipeline
In pyTerrier I have list of single terms. For example (I choose those tokens to be as relevant as possible and as irrelevant as possible to enhance the effect):
...
- 101
3
votes
2
answers
615
views
Gensim doc2vec error: KeyError: "word 'senseless' not in vocabulary"
I am new to machine learning and tried doc2vec on quora duplicate dataset. new_dfx has columns 'question1' and 'question2' which has preprocessed questions in each row. Following is the tagged ...
0
votes
1
answer
87
views
Gensim: create a dictionary from a large corpus without loading it in RAM?
The topic modelling library Gensim offers the ability to stream a large document instead of storing it in memory.
Streaming is possible for the stage of converting the corpus to BOW, but the ...
- 25k
1
vote
0
answers
257
views
How can I decide the threshold value for relevance score in a search problem?
I am using a LSA/TF-IDF/BM25/Ensemble models for text search and finally calculating similarity score to rank my search. I would like to decide a threshold value for the score, below which I would not ...
0
votes
0
answers
43
views
Are the word of women and men different when expressing their views on the same subject?
My data includes women's comments on X and Y and men's comments on X and Y. Each comment is of equal length. I will calculate how much different the word choice between men and women when commenting ...
- 9
1
vote
1
answer
20
views
Recommend products based on historical queries of other users
Given the user data as in the following:
...
- 755
-1
votes
2
answers
2k
views
How to fit Word2Vec on test data?
I am working on a Sentiment Analysis problem. I am using Gensim's Word2Vec to vectorize my data in the following way:
...
- 1,941
1
vote
2
answers
4k
views
How to calculate the mean average of word embedding and then compare strings using sklearn.metrics.pairwise
I am totally new to this topic, that's why I am so confused or stuck in this code for a while, but I am not sure how to solve it correctly.
My goal is to write a short text embedding using vector ...
- 11
2
votes
2
answers
627
views
How to examine if a Doc2Vec model is sufficiently trained?
I started experimenting with gensim's Doc2Vec for sentiment analysis. For the training of the embedding itself, I have seen examples using a reduced learning rate with a few 10s or even a few hundred ...
- 131
0
votes
1
answer
206
views
Training fasttext on your own corpus
I want to train fasttext on my own corpus. However, I have a small question before continuing. Do I need each sentences as a different item in corpus or can I have many sentences as one item?
For ...
- 113
1
vote
1
answer
216
views
Why do we calculate the vector of a document by averaging the vectors of all the words?
I am trying to build a search engine to query a folder of documents. Tutorials online suggest that we should obtain the vector of a document by averaging the vectors of all the words, then compare ...
- 51
1
vote
0
answers
53
views
default estimation method of gensim's word2vec skipgram?
I am now trying to use word2vec by estimating skipgram embeddings via NCE (noise contrastive estimation) rather than conventional negative sampling method, as a recent paper did (https://asistdl....
- 11
4
votes
2
answers
1k
views
Does spaCy support multiple GPUs?
I was wondering if spaCy supports multi-GPU via mpi4py?
I am currently using spaCy's nlp.pipe for Named Entity Recognition on a high-performance-computing cluster ...
- 163
2
votes
1
answer
3k
views
Fine-tuning pre-trained Word2Vec model with Gensim 4.0
With Gensim < 4.0, we can retrain a word2vec model using the following code:
...
- 51
0
votes
0
answers
35
views
LSA Model Improvement
I followed gensim's Core Tutorial and build an LSA Classification, topic modeling and Document Similarity model for newsgroups dataset.
My code is available here.
I need help with below 3 concepts.
...
- 1
2
votes
2
answers
283
views
classification of similar text input features with text output label
I hope somebody can provide guidance/input/advice on my project, where I believe AI can help.
I have a general understanding of AI, but I lack a formal training.
I've never built a neural net from ...
- 73
2
votes
0
answers
41
views
Evaluate Topic Modelling on synthetic data
I try to find the optimal number of topics on a synthetic corpus (so a list of lists of tokens I generate using various parameters). I, therefore, know the true number of topics and the true topics ...
- 21
5
votes
2
answers
1k
views
Why is averaging the vectors required in word2vec?
While implementing word2vec using gensim by following few tutorials online, one thing that I couldn't understand is the reason why word vectors are averaged once the model is trained. Few example ...
- 163
2
votes
0
answers
259
views
Text preprocessing on corpus in pipeline before Gensim word2vec training
I have a large compressed corpus, about 30gb in .txt.gz format. In raw format it can be used as input to word2vec like this:
...
1
vote
1
answer
331
views
LDA topic model has 0-weight topics, is that normal?
While experimenting with different number of topics for the Gensim implementation of LDA, I found that for a high number of topics, the output often consists of topics with all weights equal to zero. ...
- 179
1
vote
1
answer
20
views
Watch list of Tweets with unknown model
I have a pre-trained model that I load after import gensim using model = KeyedVectors.load_word2vec_format('path', binary = True)...
0
votes
1
answer
2k
views
Get most likely topic per document in pandas dataframe using gensim
I am using gensim LDA to build a topic model for a bunch of documents that I have stored in a pandas data frame. Once the model is built, I can call ...
- 167
1
vote
0
answers
62
views
doc2vec - paragraph or article as document
I'm trying to train a doc2vec model on the German wiki corpus. While looking for the best practice I've found different possibilities on how to create the training data.
Should I split every Wikipedia ...
- 143
2
votes
1
answer
1k
views
How to test the quality of a word embedding?
I have trained a word2vec model using GenSim 4.
The problem is that my corpus is quite small.
How can I test the quality of the word embeddings I have obtained?
Is there some standard measures to do ...
- 196
0
votes
1
answer
561
views
Difference between Word Embedding and Text Embedding
I am working on a dataset of amazon alexa reviews and wish to cluster them in positive and negative clusters. I am using Word2Vec for vectorization so wanted to know the difference between Text ...
- 51
5
votes
1
answer
2k
views
How to choose threshold for gensim Phrases when generating bigrams?
I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA
...
- 151
0
votes
2
answers
1k
views
Error in using sklearn's GridSearchCV on Word2Vec
I am using the sklearn_api of gensim to create an estimator for a Word2vec model to pass it to sklearn's gridsearch . My code is as follows :
...
- 277
0
votes
1
answer
295
views
Convert bin model to pickle [closed]
I trained a word2vec model using Gensim library which is of type .bin
Q1: can we convert this trained model in bin format to pickle?
Q2: would it speed up the execution time?
- 167
1
vote
1
answer
217
views
Two questions about word2vec and gensim
I've written the code below to try word2vec implementation of gensim. I've two questions:
Even though I've removed stop words, the word "the" is listed as one of the most similar words of &...
- 123
2
votes
3
answers
229
views
How to identify text similarity based on training data?
I have a set of documents (1 to 11) for which the labeling is done.
Lets Assume:
...
- 129
3
votes
1
answer
155
views
How to work with different Encoding for Foreign Languages
I've got a Word Embedding File called model.txt. This contains 100 Dimensional vectors for over a million French words. These words contain accented characters such ...
- 529
1
vote
0
answers
578
views
Extracting vectors of FastText own model to use it on a NN
I have trained my own model of fasttext using the pretrained model of English available on their website with the next code:
...
- 111
4
votes
1
answer
1k
views
Predicting the missing word using fasttext pretrained word embedding models (CBOW vs skipgram)
I am trying to implement a simple word prediction algorithm for filling a gap in a sentence by choosing from several options:
Driving a ---- is not fun in London streets.
Apple
Car
Book
King
With ...
- 53
2
votes
1
answer
41
views
Topic modelling on only 24 documents gives the same "topic" for any K
Description:
I have 24 documents, each one of around 2.5K tokens. They are public speeches.
My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
- 31
0
votes
1
answer
532
views
What's wrong with RF/SVM with word embedding (GloVe)?
I searched many times in google for examples on word embedding (specifically GloVe) with Random forest and I couldn't find any single example. For GloVe, it was all either LSTM or CNN. Maybe there's ...
- 65
2
votes
1
answer
113
views
Annotating the vocabulary using Word2vec model
I am trying to label the vocabulary in the corpus.
I have trained the word2vec model on the corpus
I have grouped the words which are related based on the score as key as the first word as the key ...
- 51
0
votes
0
answers
84
views
To map topic to a document after topic modeling is done with LDA
Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and ...
- 11
1
vote
1
answer
231
views
Siamese networks vs Semantic similarity (may be gensim)
I am trying to understand the Siamese networks . In this vector is calculated for an object (say an image) and a distance metric is applied (say manhatten) on two vectors produced by the neural ...
- 894
2
votes
1
answer
758
views
extract document topic vectors from lda model
how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation
- 21
8
votes
1
answer
4k
views
Difference between Gensim word2vec and keras Embedding layer
I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the ...
- 2,735
0
votes
1
answer
88
views
Models after word2vec outputs
I am originally using a bag of word (2-gram) model to approach a classification problem. The one hot encoding of the 2-gram output was sent to a logistic regression or neural network to build a ...
- 2,735
0
votes
1
answer
622
views
Length of document in doc2vec
I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired ...
- 229
8
votes
1
answer
1k
views
Gensim LDA model: return keywords based on relevance (λ - lambda) value
I am using the gensim library for topic modeling, more specifically LDA. I created my corpus, my dictionary, and my LDA model. With the help of the pyLDAvis library I visualized the results. When I ...
- 81
1
vote
1
answer
109
views
how to do topic modeling on very huge data?
When i come to know that gensim is useful library for topic modeling, I tried it on my huge amount of document. It works well only if the dictionary size is to be fix. In my case i have each and every ...
- 267
3
votes
1
answer
829
views
Metrics for unsupervised doc2vec model
I have just built a simple doc2vec model using the gensim library, pretty much followed the tutorial located here.
The methods provided for checking the quality of the model are very manual and ...
- 120
1
vote
0
answers
1k
views
How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?
I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data.
At present I ...
- 11
0
votes
1
answer
2k
views
Does gensim use Negative sampling in Word2vec?
When I train a word2vec model in Gensim on a huge amount of words/data (let's say hundreds of thousands of word vectors), is gensim using negative sampling automatically?
Alternatively, is there a ...
- 6,095