2022 Developer Survey is open! Take survey.

Questions tagged [tfidf]

tf–idf (term frequency–inverse document frequency), is a numerical statistic using in nlp that is intended to reflect how important a word is to a document in a collection or corpus. It is often used as a weighting factor in searches of information retrieval, text mining, and user modeling. tf–idf increases proportionally the number of times a word appears in the document.

Filter by
Sorted by
Tagged with
0 votes
1 answer
28 views

Optimal clusters for K-means not clear - any ideas?

I have a toy dataset of 10,000 strings of people's names, addresses and birthdays. As a quirk of the data collection process it is highly likely there are duplicate people caused by typos and I am ...
user avatar
  • 187
0 votes
0 answers
8 views

Naive Bayes TfidfVectorizer predicts everything to one class

I'm trying to run Multinomial Bayes classificator on various balanced data sets and comparing 2 different vectorizers: TfidfVectorizer and CountVectorizer. I have 3 classes: NEG, NEU and POS. I have ...
user avatar
1 vote
1 answer
24 views

NLP Basic input doubt

I actually have a basic doubt in NLP, When we consider traditional models like Decision trees, The feature column order is important, Like first column is fixed with some particular attribute. So If, ...
user avatar
  • 53
1 vote
1 answer
8 views

How to justify logarithmically scaled frequency for tf in tf-idf?

I am studying tf-idf (term frequency - inverse document frequency). The original logic for tf was straightforward: count of term t / number of total terms in the document. However, I came across the ...
user avatar
0 votes
1 answer
295 views

'list' object has no attribute 'lower' TfidfVectorizer

I have a dataframe with two text columns and I converted them to a list. I seperated the train and test data as well. But while making a base model TfidfVectorizer throws me an error of 'list' object ...
user avatar
1 vote
1 answer
14 views

Document Similarity with User Preference

To measure the similarity between two documents, one can use, e.g. TF-IDF/Cosine Similarity. Supposing that after calculating the similarity scores of Doc A against ...
user avatar
1 vote
1 answer
67 views

What are the exact differences between Word Embedding and Word Vectorization?

I am learning NLP. I have tried to figure out the exact difference between Word Embedding and Word Vectorization. However, seems like some articles use these words interchangeably. But I think there ...
user avatar
  • 13
0 votes
0 answers
53 views

Input 0 of layer max_pooling1d_3 is incompatible with the layer Error

Ok, so basically, i have some Tf-Idf features and some additional features like wordcount, sentiment on my data. Now, according to my knowledge, when we use Convolutional layer, the data needs to be ...
user avatar
1 vote
1 answer
23 views

Distinguishing text with opposite meanings in SVM (False Information Detection)

I am currently working on a Binary Text Classification Model (False Information Detection) using Support Vector Machine and used TF-IDF as text vectorizer in Python. I have already tried training the ...
user avatar
0 votes
1 answer
59 views

How to have a fixed no of features for input layer of a neural network when using TF-IDF

So basically my question is hypothetically lets say: I have a column containing 2000 rows of texts, and when I apply tf-idf, I get 27 features like shown below. Now once I do that, I could consider ...
user avatar
2 votes
1 answer
288 views

My custom stop-words list using tf-idf

I want to make my own stop words list, I computed tf-idf scores for my terms. Can I consider those words highlighted with red to be stop word? and what should my threshold be for stop words that ...
user avatar
  • 45
0 votes
0 answers
18 views

How to find similar document (using gensim) given two or more other documents?

I am developing a similarity program to compare documents, and I’ve successfully trained my model with Gensim (TFIDF and LSI) in order to compare two documents of each other, and it works great. I can ...
user avatar
  • 101
0 votes
0 answers
35 views

How to unify weights in my dataset

I have a symptom-disease network that consists of four attributes: symptom, disease, co-occurrence and TF-IDF. I'm considering the TF-IDF attribute as the weight of my network edges and symptom and ...
user avatar
0 votes
0 answers
21 views

DBSCAN getting one huge cluster with noisy points

I'm currently trying to cluster customer service email answers (NLP). When I use DBSCAN with TF-IDF embeddings + Annoy indexes, I get good clusters. But, when I use DBSCAN with FastText embeddings + ...
user avatar
0 votes
0 answers
8 views

Natural Language gender classification task with very small training set

The task involving determining the gender of the creator of a Reddit post. Given a post and its title, I need a model to output a probability vector $[p_{male},p_{female}]$. The difficulty here is ...
user avatar
2 votes
1 answer
43 views

Why tfidf of one document is not zero?

I'm new to nlp. Recently I wanted to do little nlp tasks, and faced strange thing. That is I have run the following code ...
user avatar
  • 123
0 votes
2 answers
375 views

What is the best way to limit number of features in TF-IDF?

I am using the tf-idf to build representations. It is large dataset and it quickly becomes too much for my RAM if I convert the matrix to a Data-Frame. What is the best way to reduce the number of ...
user avatar
0 votes
0 answers
22 views

Keyword Imputation Using Machine Learning and NLP

I am fairly new to machine learning and data imputation so forgive me if I am using wrong words or not feasible ideas. I have a table as follows, with text in the Headline column and a list of ...
user avatar
0 votes
2 answers
27 views

Encoding feature containing both text and string?

I have a feature which has following entries:- | Exterior | | -------- | | Vinyl | | Wd Sdng | | MetalSd | | Wd Sdng | | HdBoard | | BrkFace | | Wd Sdng | ...
user avatar
  • 1,241
0 votes
0 answers
82 views

How to measure the pairwise similarity between two textual data sets?

I have N textual data sets, and each one is composed of thousands of documents. I want to compare them to find which data sets are more similar (Similar to what it ...
user avatar
  • 109
2 votes
1 answer
42 views

Using word embeddings as features in classification algorithms?

I see there are ways to combine word vectors to form documents by taking averages or weighted averages. However, as a result of averaging there is a loss of information. Are there ways to retain the ...
user avatar
2 votes
0 answers
61 views

Can I rescale TF matrix or TF-IDF matrix using StandardScaler prior to Logisitc Lasso regression?

I am trying to use Logistic Lasso to classify documents as 1 or 0. I've tried using both the TF matrix and TF-IDF matrix representations of the documents as my predictors. I've found that if I use the ...
user avatar
0 votes
4 answers
576 views

Creating & handling large matrices in python? [closed]

I need to create a large matrix of size 400,000*400,000 and do some transformation on it. I am not able to do it using python in my laptop due to memory constraints. What technologies I can use to ...
user avatar
0 votes
1 answer
124 views

TF-IDF for 400,000+ unique words in corpus?

I have a corpus with over 400,000 unique words. I would like to build a TF-IDF matrix for this corpus. I have tried doing this on my laptop (16GB RAM) and Google Colab, but am unable to do so due to ...
user avatar
0 votes
0 answers
12 views

How can I use Ensemble learning of two models with different features as an input?

I have a fake news detection problem and it predicts the binary labels "1"&"0" by vectorizing the 'tweet' column, I use three different models for detection but I want to use ...
user avatar
1 vote
0 answers
56 views

Running PCA on top of tf-idf features?

Is it a good idea to run PCA on top of attributes obtained with Tf-Idf? The tf-idf returns a lot of attributes so in that case I believe it is a good idea to run PCA to reduce the number of dimensions....
user avatar
0 votes
2 answers
100 views

How do I get ngrams for all combinations of words in a sentence?

Lets say I have a sentence "I need multiple ngrams". If I create bigrams using Tf idf vectorizer it will create bigrams only using consecutive words. i.e. I will get "I need", &...
user avatar
0 votes
0 answers
53 views

Sparse matrix after vectorization giving size = 1

I am working on a NLP problem https://www.kaggle.com/c/nlp-getting-started. I want to perform vectorization after train_test_split but when I do that, the resulting ...
user avatar
  • 1,241
0 votes
0 answers
10 views

TF-IDF to find technical terms

I have some sentences and I want to see whether or not they contain words that are technical terms. I was thinking of working with Wikipedia texts: finding the most common words in a certain article, ...
user avatar
1 vote
1 answer
230 views

tf-idf for sentence level features

Many papers mention comparing sentences using the tf-idf metric, e.g. Paper. They state: The first one is based on tf-idf where the value of the the corresponding dimension in the vector ...
user avatar
  • 113
1 vote
1 answer
204 views

How to match a corpus with a string of words using a TF-IDF matrix?

I am trying to match strings of words with a website that has bulletpoints whose text is most similar to it. The way I thought of doing it is to get all of the documents from each bulletpoint into one ...
user avatar
  • 133
1 vote
0 answers
99 views

Hashing trick for dimensionality reduction

I am building a model that uses TF-IDF NLP features in Spark Mllib. The TF-IDF HashingTF function in Mllib uses the 'hashing trick' to efficiently allocate terms to features. My question is: does the ...
user avatar
  • 11
0 votes
0 answers
15 views

How does L1 normalisation work in Binary Classification?

I was working on a project where I was using TF*IDF algorithm. After applying grid search, I got the tfidf_norm=l1. Can someone explain how L1 normalisation form works in binary classification?(I have ...
user avatar
3 votes
1 answer
140 views

How to apply TFIDF in structured dataset in Python?

I know that TFIDF is an NLP method for feature extraction. and I know that there are libraries that calculate TFIDF directly from the text. This is not what I want though In my case, my text dataset ...
user avatar
  • 497
1 vote
0 answers
187 views

How does sklearn's tf-idf vectorizer pick the bigrams and trigrams?

...
user avatar
1 vote
2 answers
422 views

How to decide to go with BOW or TFIDF

I know that there are methods that help in selecting features such as Matual Info, and Info Gain, etc. But for datasets with thousands of records and thousands of features it is time consuming to ...
user avatar
  • 497
1 vote
0 answers
64 views

Matching documents from different sets with tfidf and cosine distance

I have two different set of documents S1, S2, with 30 text documents each. Using some text representation method, such as tfidf ...
user avatar
1 vote
3 answers
615 views

How to create a big data frame in Python

I have a sparse matrix, $X$, created by TfidfVectorizer and its size is $(500000, 200000)$. I want to convert $X$ to a data frame but I'm always getting a memory error. I tried ...
user avatar
1 vote
1 answer
25 views

Integer encoding and weighing when one feature consists of more names [closed]

Hello I am trying to make a content based movie recommendation system and one feature is genre of the movie. I will give an integer number to each genre randomly. However, some movies are of more than ...
user avatar
  • 15
2 votes
1 answer
22 views

When does it make sense to add numbers with different units?

Given two vectors containing numbers that have different natures / units, (example length in Meters and weight in Kilograms), does it make sense to calculate euclidean distance between these two ...
user avatar
1 vote
1 answer
50 views

Classification using texts as features

I want to build a classification model to match customers and products. I have a description of each product, and a description of each customer, and the label : ...
user avatar
0 votes
2 answers
82 views

How to decide which method to use TFIDF, or BOW

In a huge dataset for NLP it is taking very long time to classify my dataset therefore, trying each feature extraction method separetly is time consuming and not effecient. Is there a way that can ...
user avatar
  • 497
0 votes
2 answers
179 views

Why is the idf important in tf-idf when it seems to just re-scale your features?

I am trying to understand why tf-idf is useful. As I understand the formula to work out the tf-idf is: Can someone explain what is wrong with the reasoning below: Imagine I have 100 documents that ...
user avatar
0 votes
0 answers
24 views

Which would be an ideal model to get a specific sub string from a bigger string?

I have a corpus of documents whose some lines have information like this: wt 210 1b 14.4 oz (98 kg) or ...
user avatar
0 votes
1 answer
288 views

Why does using a standard scalar on my tf idf matrix make it perform better?

I have a TF-IDF matrix transformed on a list of tweets from a data set I am using. I have a pipeline where I initiate a StandardScalar and then next have my SVM with a linear kernel and auto gamma as ...
user avatar
  • 1
1 vote
1 answer
133 views

Comparing TFIDF vectors of different shapes

I'm working on a project using TF-IDF vectors and agglomerative clustering -- the idea is that the corpus of documents increases over time, and when a new document is added, the mean cosine similarity ...
user avatar
1 vote
1 answer
610 views

Why I would use TF-IDF after Bag-of-Words (CountVectorizer)?

In my recent studies over Machine Learning NLP tasks I found this very nice tutorial teaching how to build your first text classifier: https://towardsdatascience.com/machine-learning-nlp-text-...
user avatar
0 votes
1 answer
335 views

SKLEARN GridSearchCV hinting higher accuracy than Pipeline but with same parameters as Pipeline estimators

I have pipeline estimators like this: ...
user avatar
  • 133
1 vote
0 answers
34 views

Text Analysis : Recommendation to identify cause of loss from claim narrative documents

I am trying to analyze auto claims narrative documents which contain description about the accident usually free text written by claims executives. Is there a nlp technique I could use to identify ...
user avatar
  • 271
0 votes
1 answer
177 views

Ordering of standardization, pca, and/or tfidf for neural network

I have 60k rows of text data. I have tokenized it into 55k columns. I am using a neural network to classify the data but have some questions about how to order my preprocessing steps. I have too much ...
user avatar
  • 3