Questions tagged [tfidf]

tf–idf (term frequency–inverse document frequency), is a numerical statistic using in nlp that is intended to reflect how important a word is to a document in a collection or corpus. It is often used as a weighting factor in searches of information retrieval, text mining, and user modeling. tf–idf increases proportionally the number of times a word appears in the document.

Filter by
Sorted by
Tagged with
14
votes
2answers
28k views

Word2Vec embeddings with TF-IDF

When you train the word2vec model (using for instance, gensim) you supply a list of words/sentences. But there does not seem to be a way to specify weights for the words calculated for instance using ...
5
votes
3answers
5k views

Weighted sum of word vectors for document similarity

I have trained a word2vec model on a corpus of documents. I then compute the term frequency (the same Tf in TfIDF) of each word in each document, multiply each words Tf by its corresponding word ...
0
votes
0answers
19 views

Sparse matrix after vectorization giving size = 1

I am working on a NLP problem https://www.kaggle.com/c/nlp-getting-started. I want to perform vectorization after train_test_split but when I do that, the resulting ...