Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more

Questions tagged [tfidf]

tf–idf (term frequency–inverse document frequency), is a numerical statistic using in nlp that is intended to reflect how important a word is to a document in a collection or corpus. It is often used as a weighting factor in searches of information retrieval, text mining, and user modeling. tf–idf increases proportionally the number of times a word appears in the document.

Filter by
Sorted by
Tagged with
0 votes
0 answers
24 views

I want to make a Career suggestion model

There is a dataset having job titles and the descriptions. when a person enter his skills i need to output which category of job he should do. i have already created that using cosine similarity.(If ...
pycoder's user avatar
0 votes
0 answers
11 views

Why does TF-IDF work in TfidfVectorizer?

As I understand TF-IDF, the IDF value of the word "art" = log_e(3/1) + 1 because there are 3 documents in the data set and the word "art" appears once. But after I use the print ...
Khang Khang's user avatar
0 votes
0 answers
17 views

Why do my tf-idf values not appear consistent?

I have a series of tweets that I've converted to tokens. Among them are the following: geraldkutney happen realize happen conveniently rename catch yet emergency post fact come government ...
Daniel V's user avatar
  • 101
2 votes
1 answer
67 views

Text preprocessing decreases classifier accuracy

I try to solve a binary text classification problem using sklearn's Tfidf Vecotrizer and a naive bayes classifier. Before I pass the training/test data to the vectorizer I do some text preprocessing. ...
MC Racoon's user avatar
0 votes
1 answer
90 views

Text classification with very short strings

I have a dataset of short job titles (e.g., 'marketing manager', 'system administrator', etc.) and their respective Census occupation code (e.g., 1006 Computer systems analysts). I am interested in ...
topi-dont-know's user avatar
1 vote
0 answers
29 views

How to implement Ant Lion Optimization (ALO) feature selection for KNN Classification Problems?

I have been assigned for a project related to text data classification, i have preprocess and vectorized the data with TF-IDF. For feature selection i am using pyMetahueristic library to implement ALO ...
Physics69's user avatar
0 votes
0 answers
59 views

Doing LDA (NLP task): Does it make sense to use tf-idf vectors?

Most implementations that I've seen of LDA seem to use simple word counts when giving the document-term frequency matrix. What would happen if we were to give the tf-idf matrix instead of a simple ...
An old man in the sea.'s user avatar
0 votes
1 answer
165 views

Why my sentiment analysis model is overfitting?

The task is to predict sentiment from 1 to 10 based on Russian reviews. The training data size is 20000 records, of which 1000 were preserved as a validation set. The preprocessing steps included ...
Renat Abdrakhmanov's user avatar
0 votes
1 answer
65 views

Dealing with rich vocabulary and a low average frequency of words in NLP

What is the best way to deal with a dataset that has a rich vocabulary and a low average frequency of words that is showing low validation accuracy? While reading online I saw many people recommending ...
Medhat's user avatar
  • 1
0 votes
0 answers
107 views

On which texts should TfidfVectorizer be fitted when using TF-IDF cosine for text similarity?

I wonder on which texts should TfidfVectorizer be fitted when using TF-IDF cosine for text similarity. Should TfidfVectorizer be fitted on the texts that are analyzed for text similarity, or some ...
Franck Dernoncourt's user avatar
0 votes
1 answer
221 views

how to evaluate the combination of tfidf and kmeans

For my nlp problem I'm using a combination of TFIDF and KMeans from the sklearn package. The tfidf gets the vectors and then I use Kmeans to cluster the texts based on the vectors. I have a few ...
james pow's user avatar
  • 167
2 votes
2 answers
110 views

In sklearn tfidf what is the difference between term frequecy and document frequency

Looking at the sklearn tfidf page: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html and trying to understand the difference between term frequency ...
james pow's user avatar
  • 167
0 votes
1 answer
232 views

how do you get the frequency of the terms generated by tfidf.get_feature_names_out()

After fitting with tfidf, I'm looking at the features that were generated: ...
james pow's user avatar
  • 167
1 vote
1 answer
509 views

Do Sampling before or after TFIDF step?

This is a multiclass text classification problem. The dataset has a class imbalance and I'm planning to use a sampling technique before modeling. Should the sampling be done before/after the ...
Mohith7548's user avatar
3 votes
1 answer
462 views

Is there a way to map words to their synonyms in tfidf?

I have the following code: ...
james pow's user avatar
  • 167
1 vote
1 answer
110 views

Why is max_features ordered by term frequency instead of inverse document frequency

In the docs: https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html it is explained that max_features is ordered by ...
james pow's user avatar
  • 167
1 vote
0 answers
375 views

How can I decide the threshold value for relevance score in a search problem?

I am using a LSA/TF-IDF/BM25/Ensemble models for text search and finally calculating similarity score to rank my search. I would like to decide a threshold value for the score, below which I would not ...
Prateek Coder's user avatar
1 vote
1 answer
98 views

Effectiveness of tf-idf on documents with repeated keywords

I was doing some ML reading and came upon tf-idf. The tf portion counts the relative frequency of a relevant word in a document, while idf measures how common or rare a word is across the corpus. The ...
disguisedtoast's user avatar
0 votes
0 answers
146 views

How do I use TF-IDF for set of keywords?

I have a set of keyword K = {K1,K2,K3,...} K1 = (president governor) k2 = (foot ball players goal) K3=(Hero Heroine song singer) etc. like these and each K1...Kn belongs to some category like in ...
j123's user avatar
  • 11
1 vote
1 answer
50 views

Is there a tokenizer to tokenize Swift language code in python

...
Shamsudeen McHalwai's user avatar
0 votes
0 answers
105 views

Is TF-IDF for text classification transferable between corpuses?

I am using TF-IDF for text classification and my solution works well according to the performance metric of my choice (F1 macro). To speed up the training process I have used PCA to reduce the ...
Ali Asgari's user avatar
0 votes
1 answer
42 views

Optimal clusters for K-means not clear - any ideas?

I have a toy dataset of 10,000 strings of people's names, addresses and birthdays. As a quirk of the data collection process it is highly likely there are duplicate people caused by typos and I am ...
Sandy Lee's user avatar
  • 247
1 vote
1 answer
41 views

NLP Basic input doubt

I actually have a basic doubt in NLP, When we consider traditional models like Decision trees, The feature column order is important, Like first column is fixed with some particular attribute. So If, ...
mewbie's user avatar
  • 109
1 vote
1 answer
64 views

How to justify logarithmically scaled frequency for tf in tf-idf?

I am studying tf-idf (term frequency - inverse document frequency). The original logic for tf was straightforward: count of term t / number of total terms in the document. However, I came across the ...
Fred Chang's user avatar
0 votes
1 answer
5k views

'list' object has no attribute 'lower' TfidfVectorizer

I have a dataframe with two text columns and I converted them to a list. I seperated the train and test data as well. But while making a base model TfidfVectorizer throws me an error of 'list' object ...
Tanvi Punjani's user avatar
1 vote
1 answer
59 views

Document Similarity with User Preference

To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
JoyfulPanda's user avatar
6 votes
2 answers
7k views

What are the exact differences between Word Embedding and Word Vectorization?

I am learning NLP. I have tried to figure out the exact difference between Word Embedding and Word Vectorization. However, seems like some articles use these words interchangeably. But I think there ...
Nahid 's user avatar
  • 63
1 vote
1 answer
26 views

Distinguishing text with opposite meanings in SVM (False Information Detection)

I am currently working on a Binary Text Classification Model (False Information Detection) using Support Vector Machine and used TF-IDF as text vectorizer in Python. I have already tried training the ...
alexand88r's user avatar
0 votes
1 answer
340 views

How to have a fixed no of features for input layer of a neural network when using TF-IDF

So basically my question is hypothetically lets say: I have a column containing 2000 rows of texts, and when I apply tf-idf, I get 27 features like shown below. Now once I do that, I could consider ...
Yeshan Santhush's user avatar
3 votes
1 answer
2k views

My custom stop-words list using tf-idf

I want to make my own stop words list, I computed tf-idf scores for my terms. Can I consider those words highlighted with red to be stop word? and what should my threshold be for stop words that ...
Maxi's user avatar
  • 89
0 votes
0 answers
38 views

How to unify weights in my dataset

I have a symptom-disease network that consists of four attributes: symptom, disease, co-occurrence and TF-IDF. I'm considering the TF-IDF attribute as the weight of my network edges and symptom and ...
Alireza Azhdari's user avatar
2 votes
1 answer
334 views

Why tfidf of one document is not zero?

I'm new to nlp. Recently I wanted to do little nlp tasks, and faced strange thing. That is I have run the following code ...
Doralisa's user avatar
  • 123
0 votes
2 answers
4k views

What is the best way to limit number of features in TF-IDF?

I am using the tf-idf to build representations. It is large dataset and it quickly becomes too much for my RAM if I convert the matrix to a Data-Frame. What is the best way to reduce the number of ...
Borut Flis's user avatar
0 votes
2 answers
46 views

Encoding feature containing both text and string?

I have a feature which has following entries:- | Exterior | | -------- | | Vinyl | | Wd Sdng | | MetalSd | | Wd Sdng | | HdBoard | | BrkFace | | Wd Sdng | ...
spectre's user avatar
  • 2,105
1 vote
0 answers
279 views

How to measure the pairwise similarity between two textual data sets?

I have N textual data sets, and each one is composed of thousands of documents. I want to compare them to find which data sets are more similar (Similar to what it ...
revy's user avatar
  • 119
2 votes
1 answer
135 views

Using word embeddings as features in classification algorithms?

I see there are ways to combine word vectors to form documents by taking averages or weighted averages. However, as a result of averaging there is a loss of information. Are there ways to retain the ...
user16584277's user avatar
2 votes
0 answers
255 views

Can I rescale TF matrix or TF-IDF matrix using StandardScaler prior to Logisitc Lasso regression?

I am trying to use Logistic Lasso to classify documents as 1 or 0. I've tried using both the TF matrix and TF-IDF matrix representations of the documents as my predictors. I've found that if I use the ...
Patrick Steele's user avatar
0 votes
4 answers
2k views

Creating & handling large matrices in python? [closed]

I need to create a large matrix of size 400,000*400,000 and do some transformation on it. I am not able to do it using python in my laptop due to memory constraints. What technologies I can use to ...
user16584277's user avatar
1 vote
1 answer
1k views

TF-IDF for 400,000+ unique words in corpus?

I have a corpus with over 400,000 unique words. I would like to build a TF-IDF matrix for this corpus. I have tried doing this on my laptop (16GB RAM) and Google Colab, but am unable to do so due to ...
user16584277's user avatar
1 vote
1 answer
3k views

Running PCA on top of tf-idf features?

Is it a good idea to run PCA on top of attributes obtained with Tf-Idf? The tf-idf returns a lot of attributes so in that case I believe it is a good idea to run PCA to reduce the number of dimensions....
Borut Flis's user avatar
1 vote
2 answers
747 views

How do I get ngrams for all combinations of words in a sentence?

Lets say I have a sentence "I need multiple ngrams". If I create bigrams using Tf idf vectorizer it will create bigrams only using consecutive words. i.e. I will get "I need", &...
user16584277's user avatar
0 votes
1 answer
498 views

Sparse matrix after vectorization giving size = 1

I am working on a NLP problem https://www.kaggle.com/c/nlp-getting-started. I want to perform vectorization after train_test_split but when I do that, the resulting ...
spectre's user avatar
  • 2,105
1 vote
1 answer
1k views

tf-idf for sentence level features

Many papers mention comparing sentences using the tf-idf metric, e.g. Paper. They state: The first one is based on tf-idf where the value of the the corresponding dimension in the vector ...
DsCpp's user avatar
  • 113
1 vote
1 answer
634 views

How to match a corpus with a string of words using a TF-IDF matrix?

I am trying to match strings of words with a website that has bulletpoints whose text is most similar to it. The way I thought of doing it is to get all of the documents from each bulletpoint into one ...
sangstar's user avatar
  • 133
2 votes
0 answers
394 views

Hashing trick for dimensionality reduction

I am building a model that uses TF-IDF NLP features in Spark Mllib. The TF-IDF HashingTF function in Mllib uses the 'hashing trick' to efficiently allocate terms to features. My question is: does the ...
John's user avatar
  • 21
3 votes
1 answer
382 views

How to apply TFIDF in structured dataset in Python?

I know that TFIDF is an NLP method for feature extraction. and I know that there are libraries that calculate TFIDF directly from the text. This is not what I want though In my case, my text dataset ...
asmgx's user avatar
  • 549
2 votes
0 answers
495 views

How does sklearn's tf-idf vectorizer pick the bigrams and trigrams?

...
Sachin Krishna's user avatar
1 vote
2 answers
1k views

How to decide to go with BOW or TFIDF

I know that there are methods that help in selecting features such as Matual Info, and Info Gain, etc. But for datasets with thousands of records and thousands of features it is time consuming to ...
asmgx's user avatar
  • 549
1 vote
0 answers
176 views

Matching documents from different sets with tfidf and cosine distance

I have two different set of documents S1, S2, with 30 text documents each. Using some text representation method, such as tfidf ...
lynx's user avatar
  • 113
1 vote
3 answers
980 views

How to create a big data frame in Python

I have a sparse matrix, $X$, created by TfidfVectorizer and its size is $(500000, 200000)$. I want to convert $X$ to a data frame but I'm always getting a memory error. I tried ...
user avatar