Questions tagged [gensim]

gensim is the python library for topic modelling. multi-dimensional vector representation of words or sentences which preserves semantic meaning is computed through word2vec and doc2vec models.

Filter by
Sorted by
Tagged with
2
votes
1answer
50 views

Predicting the missing word using fasttext pretrained word embedding models (CBOW vs skipgram)

I am trying to implement a simple word prediction algorithm for filling a gap in a sentence by choosing from several options: Driving a ---- is not fun in London streets. Apple Car Book King With ...
1
vote
0answers
20 views

Gensim - Running Similarity Queries example with another query result in low score

I am a newbie to NLP and gensim, and I am trying to run the tutorial (Similarity Queries) (https://radimrehurek.com/gensim/tut3.html). I can follow the example, and run the expected result. However, ...
2
votes
0answers
16 views

Topic modelling on only 24 documents gives the same “topic” for any K

Description: I have 24 documents, each one of around 2.5K tokens. They are public speeches. My text preprocessing pipeline is a generic one, including punctuation removal, expansion of English ...
0
votes
1answer
54 views

What's wrong with RF/SVM with word embedding (GloVe)?

I searched many times in google for examples on word embedding (specifically GloVe) with Random forest and I couldn't find any single example. For GloVe, it was all either LSTM or CNN. Maybe there's ...
2
votes
1answer
35 views

Annotating the vocabulary using Word2vec model

I am trying to label the vocabulary in the corpus. I have trained the word2vec model on the corpus I have grouped the words which are related based on the score as key as the first word as the key ...
0
votes
0answers
13 views

To map topic to a document after topic modeling is done with LDA

Is there any way I can map generated topic from LDA to the list of documents and identify to which topic it belongs to ? I am interested in clustering documents using unsupervised learning and ...
0
votes
0answers
8 views

Gensim Corpora: why the document reference is missing?

I'm trying to understand how the gensim corpora model works. At this page, the example at a certain point create the corpus but in inspecting the content I can see ...
0
votes
1answer
45 views

Siamese networks vs Semantic similarity (may be gensim)

I am trying to understand the Siamese networks . In this vector is calculated for an object (say an image) and a distance metric is applied (say manhatten) on two vectors produced by the neural ...
0
votes
1answer
35 views

extract document topic vectors from lda model

how can I extract document-topic matrix from LDA model and use it as input features an svm classifier? I am using gensim for implementation
2
votes
1answer
463 views

Difference between Gensim word2vec and keras Embedding layer

I used the gensim word2vec package and Keras Embedding layer for various different projects. Then I realize they seem to do the ...
0
votes
0answers
390 views

Interpreting Gensim Word2Vec Training Loss

I am using Gensim to build a Word2Vec model and identify the convergence of training loss, so that I can figure out the optimal number of iterations. For understanding this since gensim's ...
0
votes
0answers
59 views

Doc2vec doesn't use all the CPU power

I have a big dataset that trying to train with a Doc2vec model. I am working on a 8 CPU, 32GB RAM, but as I can see on the monitoring tools, it only uses about 66-67% of the CPU. I am not sure if it ...
0
votes
0answers
28 views

Load pretrained embedding model on TF-Hub to calculate Word Mover's Distance (WMD) on Gensim or spaCy

I'd like to calculate Word Mover's Distance with Universal Sentence Encoder on TensorFlow Hub embedding. I have tried the example on spaCy for WMD-relax, which loads 'en' model from spaCy, but I ...
0
votes
0answers
71 views

how to calculate coherence score in topic model

I am trying to calculate coherence score in topic modeling. I am following this Github link So there I need to use the preprocessed wiki and news. I got 3 questions: if the domain that I have ...
0
votes
1answer
15 views

Models after word2vec outputs

I am originally using a bag of word (2-gram) model to approach a classification problem. The one hot encoding of the 2-gram output was sent to a logistic regression or neural network to build a ...
0
votes
1answer
39 views

Length of document in doc2vec

I have 100 sentences that I want to cluster based on similarity. I've used doc2vec to vectorize the sentences into 20 dimensional vectors and applied kmeans to cluster them. I haven't got the desired ...
3
votes
0answers
110 views

Gensim LDA model: return keywords based on relevance (λ - lambda) value

I am using gensim library for topic modeling, more specifically LDA. I have created my corpus, my dictionary and my lda model, and with the help of pyLDAvis library I visualize the results. When I ...
1
vote
1answer
27 views

how to do topic modeling on very huge data?

When i come to know that gensim is useful library for topic modeling, I tried it on my huge amount of document. It works well only if the dictionary size is to be fix. In my case i have each and every ...
1
vote
1answer
151 views

Metrics for unsupervised doc2vec model

I have just built a simple doc2vec model using the gensim library, pretty much followed the tutorial located here. The methods provided for checking the quality of the model are very manual and ...
1
vote
0answers
491 views

How to effectively tune the hyper-parameters of Gensim Doc2Vec to achieve maximum accuracy in Document Similarity problem?

I have around 20k documents with 60 - 150 words. Out of these 20K documents, there are 400 documents for which the similar document are known. These 400 documents serve as my test data. At present I ...
0
votes
1answer
258 views

Does gensim use Negative sampling in Word2vec?

When I train a word2vec model in Gensim on a huge amount of words/data (let's say hundreds of thousands of word vectors), is gensim using negative sampling automatically? Alternatively, is there a ...
0
votes
0answers
114 views

Document similarity over years: TF-IDF Word2Vec, gensim

I have two documents one at time $t$ and the other at time $t+1$. I individually calculate the TF-IDF of both documents and store my results into a document term matrix. I can load both the document ...
0
votes
0answers
57 views

Advice for making word vectors from a custom corpus

I'm working to train custom word vectors on a corpus built from my company's support tickets (using gensim). I've made some strides in getting that corpus to consist primarily of natural language (...
0
votes
0answers
40 views

Tag words of interest by using machine learning?

I have a set of documents (around 50k), each document is a sentence long. I would like to tag each document with the words that relate to risks. For example, "dangerous", "hazard", "fatal", etc... If ...
1
vote
0answers
44 views

Copying embeddings for gensim word2vec

I wanted to see if I can simply set new weights for gensim's Word2Vec without training. I get the 20 News Group data set from scikit-learn (from sklearn.datasets import fetch_20newsgroups) and trained ...
0
votes
0answers
67 views

Oddly shaped t-SNE visualizations with Word2Vec — CBOW vs Skipgram

I've been using t-SNE to graph some Gensim Word2Vec models trained on a relatively small corpus (10 epochs). For some reason, when graphed using t-SNE, the CBOW model has a cubic-like shape, whereas ...
0
votes
0answers
100 views

Doc2vec model.docvecs giving varying output

I am using doc2vec to vectorize input text. I am converting my input dataset to tagged data and giving it as input. Initially I tried with a data of 27 input text: ...
0
votes
0answers
41 views

Linking LDA topics to the input documents

I am new to LDA topic modelling. I am using gensim and am able to generate topics that make sense. Using 25k of documents, I can also print them using print_topics. ...
2
votes
1answer
330 views

word2vec word embeddings creates very distant vectors, closest similarity is still very far

I started using gensim's FastText to create word embeddings on a large corpus of a specialized domain (after finding that existing open source embeddings are not performing well on this domain), ...
1
vote
0answers
31 views

How to create pretrained word embedding text file with additional word features

I've had an idea for using word features to improve the quality of neural machine translation. Now, I would like to create word embeddings with additional word features such as pos tag, named entity, ...
0
votes
0answers
45 views

Should my LDA topic model be skewed towards only one topic? If not then how can I un-skew it?

I'm building LDA topic models in to apply against a collection of small texts and regardless of the number of topics, I'm finding that there is always one topic that is very large (in terms of ...
6
votes
3answers
4k views

Word2Vec how to choose the embedding size parameter

I'm running word2vec over collection of documents. I understand that the size of the model is the number of dimensions of the vector space that the word is embedded into. And that different dimensions ...
3
votes
3answers
1k views

How to train an existing word2vec gensim model on new words?

According to gensim docs, you can take an existing word2vec model and further train it on new words. The training is streamed, meaning sentences can be a generator, reading input data from disk ...
0
votes
0answers
36 views

How would I use Word2Vec model to find similar terms so that I can implement semantic search in some sense

I have built the model from the corpus but the problem is the similar words coming from the model is not expected. Also, This may be a broad question but I really cannot find a source where the ...
2
votes
1answer
1k views

Sentence similarity using Doc2vec

I have a list of 50k sentences such as : 'bone is making noise', 'nose is leaking' ,'eyelid is down' etc.. I'm trying to use Doc2Vec to find the most similar sentence from the 50k given a new ...
1
vote
1answer
81 views

How PV-DBOW works

The authors of the Paragraph Vector paper describe PV-DBOW with: 2.3. Paragraph Vector without word ordering: Distributed bag of words The above method considers the concatenation of the ...
1
vote
0answers
72 views

Can I use Gensim doc2vec model for classification new documents?

Can I use Gensim doc2vec model for classification new documents via infer_vector? All my tests gave too bad results, even for big datasets (10GB utf-8 texts)...
1
vote
1answer
502 views

Fasttext error while loading wiki pre-trained data

I am loading the model using gensim package this way: from gensim.models import FastText model = FastText.load_fasttext_format('wiki-news-300d-1M-subword.bin') ...
1
vote
0answers
60 views

Doc2Vec for dataset with several text fields: concatenate or separate models?

I have a dataset with several fields: description, name, header. I want to train doc2vec out of it, so that I could use vectors for classification. So I wonder, ...
0
votes
1answer
51 views

Understanding word2vec vectors representation

I'm trying to obtain the word2vec representation of few words using gensim. At present, this is the model that I have: ...
2
votes
1answer
539 views

Updating Google News Word2vec Word Embedding?

Is it possible to update the Google News Word Embedding with a custom text dataset (text data pertaining to a particular domain) ? Google News Word2Vec - Word Embedding clearly helps us to come with ...
1
vote
0answers
114 views

Can we use doc2vec to detect outlier documents?

I have a set of documents and I want to identify and remove the outlier documents. I am just wondering if doc2vec can be used for this task. Or are there any recently evolved, promising algorithms ...
2
votes
2answers
1k views

can I use public pretrained word2vec, and continue train it for domain specific text?

I have a set of reviews from apparel domain, about 100K reviews (2M words). And I want to train word2vec to do some cool NLP staff with it. However the size is not enough for creating adequate ...
2
votes
1answer
646 views

How to count number of word embeddings in Gensim Word2Vec model

I am trying to create a Word2Vec model of the the Pub Med Central corpus using the Gensim library and want to limit the total number of word embeddings to around 1 billion. I have searched high and ...
1
vote
1answer
903 views

how to update the pre-trained word2vec model with new train data using genism

Hi I have used the genism to load the Spanish fasttext word2vec model with following code: ...
0
votes
1answer
203 views

what does this doc2vec based ML predict?

I'm trying to understand what does this ML program - which based on doc2vec - predict: ...
1
vote
0answers
61 views

How is determined the context's dimension in Doc2Vec?

I would like to know how is determined the dimension of the context in Gensim Doc2Vec.
1
vote
1answer
560 views

How to compare the topic coherence between models of different number of topics?

If I'm not mistaken, in this paper here http://svn.aksw.org/papers/2015/WSDM_Topic_Evaluation/public.pdf it appears that topics with larger number of topics will inherently have larger coherence ...
1
vote
2answers
1k views

Sub topics with Latent Dirichlet Allocation

I'm training an LDA model with gensim's LdaMulticore. The topics look great, but knowing the domain I know there exists topics within topics but I'm not quite sure the best way to model this. I've ...
3
votes
2answers
2k views

When to use different Word2Vec training approaches?

So I am learning Word2Vec for the first time and my question is quite basic: How to know what approach to use? I.e, Word2Vec in Tensorflow or Word2Vec trained with Gensim ? In what cases would ...