Hot answers tagged

12 votes

Word2Vec embeddings with TF-IDF

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document ...
user avatar
  • 840
12 votes
Accepted

Using TF-IDF with other features in scikit-learn

scikit-learn's FeatureUnion concatenates features from different vectorizers. An example of combining heterogeneous data, including text, can be found here.
user avatar
10 votes
Accepted

What is the difference between a hashing vectorizer and a tfidf vectorizer

The main difference is that HashingVectorizer applies a hashing function to term frequency counts in each document, where ...
user avatar
  • 1,558
9 votes

Word2Vec embeddings with TF-IDF

Train a tfidfvectorizer with your corpus and use the following code: tfidf = Tfidfvectorizer () dict(zip(tfidf.get_feature_names(), tfidf.idf_))) Now you have a ...
user avatar
8 votes
Accepted

Should I rescale tfidf features?

The most accepted idea is that bag-of-words, Tf-Idf and other transformations should be left as is. According to some: Standardization of categorical variables might be not natural. Neither is ...
user avatar
  • 3,260
7 votes
Accepted

Weighted sum of word vectors for document similarity

Yes, your method is valid and it has been studied before it is known as Mean of Word Embeddings (MOWE) or Sum of Word Embeddings (SOWE), although your method is more a weighted mean of vectors. I ...
user avatar
  • 2,156
6 votes

What is the difference between a hashing vectorizer and a tfidf vectorizer

The HashingVectorizer has a parameter n_features which is 1048576 by default. When hashing, ...
user avatar
  • 160
5 votes

Using TF-IDF with other features in scikit-learn

Usually, if possible, you'd want to keep your matrice sparse as long as possible as it saves a lot of memory. That's why there are sparse matrices after all, otherwise, why bother? So, even if your ...
user avatar
5 votes

TF-IDF for 400,000+ unique words in corpus?

Without knowing your domain one cannot comment whether this is an appropriate size of feature names or not. However, consider this. Wordnet has database contains 155 327 words organized in 175 979 ...
user avatar
5 votes
Accepted

Why tfidf of one document is not zero?

It is because, by default sklearn's TF-IDF vectorizer will normalize the results. See the the Tf-IDF Term Weighting section of the User Guide. For your example, ...
user avatar
5 votes
Accepted

My custom stop-words list using tf-idf

There's no standard definition of stop-word, but in general stop words are very frequent words which don't contribute to the meaning of the text, like determiners, pronouns, etc. Importantly stop-word ...
user avatar
  • 21.8k
4 votes
Accepted

TF-IDF Features vs Embedding Layer

It is common for TFIDF to be a strong model. People constantly get high places in Kaggle competitions with TFIDF models. Here is a link to the winning solution that used TFIDF as one of its features (...
user avatar
  • 1,179
3 votes

Sklearn tfidf vectorize returns different shape after fit_transform()

What is wrong with your implementation is that you are passing a dataframe directly to tfidf vectorizer. If you check your data, it would look like this - ...
user avatar
  • 1,729
3 votes
Accepted

TF-IDF not a strong measure in this senario?

Weighting scheme 2 in table Recommended TF-IDF weighting schemes in tf–idf | Wikipedia should solve your problem.
user avatar
  • 46
3 votes

Are stopwords helpful when using tf-idf features for document classification?

From the way the TfIdf score is set up, there shouldn't be any significant difference in removing the stopwords. The whole point of the Idf is exactly to remove words with no semantic value from the ...
user avatar
3 votes
Accepted

Word2Vec and Tf-idf how to combine them

Word2Vec algorithms (Skip Gram and CBOW) treat each word equally, because their goal to compute word embeddings. The distinction becomes important when one needs to work with sentences or document ...
user avatar
3 votes
Accepted

TF-IDF for Topic Modeling

Formally the problem of topic modelling is a clustering problem: given a collection of text documents, group together the documents which are topically similar. So technically it can indeed be done ...
user avatar
  • 21.8k
3 votes

How to create a big data frame in Python

You can use pandas.Dataframe.sparse.from_spmatrix. It will create a Dataframe populated by pd.arrays.SparseArray from a scipy ...
user avatar
  • 2,309
3 votes

How to apply TFIDF in structured dataset in Python?

You could use pandas pivot_table() to transform your data frame into a count matrix, and then apply ...
user avatar
3 votes

Creating & handling large matrices in python?

I don't know if this is your job work or your personal project but cloud services can help you. Create a free account on Azure (you get 200$ worth free credit for 1 month) and the account is free for1 ...
user avatar
  • 1,261
3 votes

Creating & handling large matrices in python?

You can also buy some GPU's which will always help you to make up for the low memory allocation. Cloud services will help as well but the variable costs are too high if your objective is to work on ...
user avatar
3 votes
Accepted

What is the best way to limit number of features in TF-IDF?

Generally speaking the correct representation on td-idf encoding is a hyperparameter to be optimized. As suggested in the above's answers, you can go for the ...
user avatar
  • 2,191
2 votes
Accepted

Idf values of English words

The list of 20,000 most common words in English is avaiable here. By using Zipf's law, we can obtain the probability of these words as below. Zipf's Law In the English language, the probability of ...
user avatar
2 votes

Idf values of English words

I don't believe that there are any precalculated idf values out there. Inverse Document Frequency (idf) is the inverse of the number of documents in which a particular word appears in your corpus. If ...
user avatar
2 votes

Online news classification

We do something similar for financial news classification which I suspect is similar to what you're trying to do, the problem we hit when using completely automated classification was there are a ...
user avatar
  • 121
2 votes

Online news classification

The usual rule is to ask only one question per post. I will answer your first question. Build a dictionary, once, in advance. With a bit of effort you should be able to construct a dictionary that ...
user avatar
  • 2,988
2 votes

Sklearn tfidf vectorize returns different shape after fit_transform()

I had a similar problem. What I was doing was this: I loaded a pretrained tfidf vectorizer. And on the new data to be predicted, I called vectorizer.fit_transform() ...
user avatar
2 votes

Algorithm for document retrieval in QA system

I am still not quite sure how you are solving in the problem with tf-idf for a QA system. However, there have been lots of improvements that has been done in the QA domain over the years, with the ...
user avatar
2 votes
Accepted

TS-SS and Cosine similarity among text documents using TF-IDF in Python

Normalised vectors have magnitude 1, so it doesn't matter if you explicitly divide by the magnitudes or not. It's mathematically equivalent either way. I see no reason that you couldn't use ...
user avatar
  • 3,780
2 votes
Accepted

How to choose the best parameter values for TfidfVectorizer in sklearn library?

I think these parameters are mostly used when you combine the vectorizer and a machine learning model in a pipeline. Therefore, you should tune these parameters based on the outcome of your model ...
user avatar
  • 106

Only top scored, non community-wiki answers of a minimum length are eligible