Questions tagged [tfidf]

The tag has no usage guidance.

41 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3
votes
1answer
236 views

TF-IDF vs TF for classification

Let's suppose that I have a dataset of 1000 documents. Each document is a restaurant review (so relatively short text) and it has labels {Negative, Indifferent, Positive}. Let's suppose that the ...
2
votes
1answer
24 views

Text vectorizer that capture feature offset in the text?

I'm using sklearn Tfifdfvectorizer to extract feature from text towards text classification. I believe the information I need tends to be in the beginning of the document, so I would like to somehow ...
2
votes
0answers
316 views

Mixing Textual Data and Numerical Data (Neural Network)

I have 2 "nature" of data (more actually if I count images data) : Textual (that I treat with special tokenization and a TfIdfVectorizer) ~ 5000 features Non textual (like length of sentences, # of ...
2
votes
0answers
433 views

DBSMOTE on Short Text Classification

I am trying to use DBSMOTE(Density-Based Synthetic Oversampling TEqnique) to on a data set of short text--tweets to be specific. This will be used to train a classifier model in a multiclass ...
2
votes
0answers
647 views

TF-IDF Augmented Frequency vs Cosine Normalization

I am using TF-IDF for text classification and have been curious about the following two concepts. The augmented term frequency which is basically used for weighting in order to eliminate the bias ...
1
vote
1answer
36 views

Classification using texts as features

I want to build a classification model to match customers and products. I have a description of each product, and a description of each customer, and the label : ...
1
vote
0answers
15 views

Which would be an ideal model to get a specific sub string from a bigger string?

I have a corpus of documents whose some lines have information like this: wt 210 1b 14.4 oz (98 kg) or ...
1
vote
0answers
24 views

Text Analysis : Recommendation to identify cause of loss from claim narrative documents

I am trying to analyze auto claims narrative documents which contain description about the accident usually free text written by claims executives. Is there a nlp technique I could use to identify ...
1
vote
1answer
25 views

How best to embed large and noisy documents

I have a large corpus of documents (web pages) collected from various sites of around 10k-30k chars each, I am processing them to extract relevant text as much as possible, but they are never perfect. ...
1
vote
0answers
21 views

Topic alignment / topic modelling

What is the most efficient method for detecting whether the article is mostly about a specific topic, but without lots of data for training? My task is to determine how much a document is e.g. about ...
1
vote
0answers
21 views

Should I create a tfidf on a subset of a data set or use the whole corpus?

My goal in this project is to see if businesses on a list are currently customers within my organization. One piece of this involves producing a similarity score using cosine similarity on the names ...
1
vote
0answers
46 views

Semi-Supervised Learning using NLP

I am working on a drug reaction problem in which I need to extract tweets and label the tweets (binary-reaction due to drug or not). But since I don't have domain knowledge, and clustering would also ...
1
vote
3answers
41 views

Vectorize One line text data

How to vectorize one-line text data? I have used tf-idf including bigrams and trigrams but I am not able to get good results. I have purchase order descriptions which are one-liners and I need to ...
1
vote
0answers
66 views

Setting a threshold for tfidf

Let's say I have streaming textual data coming in. I run named entity recognition software to capture the entities, then I score them using tf-idf. Tf-idf scores are unbounded positively. How can I ...
1
vote
1answer
34 views

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them?

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them? That is, to get results which say that, for example, the ...
1
vote
0answers
154 views

SVM/Naive Bayesian text classification on multiple features

I was building a text classifier which takes into account certain features of the text and classifies them into two - "Yes" or "No". I have trimmed the text, removed stopwords and have applied TFIDF ...
1
vote
3answers
722 views

How do I use TF*IDF scores for my machine learning model?

I have applied TF*IDF on the 'Ad-topic line' column of my dataset. For every ad-topic line, I get the same output: Firstly, I am unable to make sense of the output. The TF*IDF values are mentioned to ...
1
vote
0answers
141 views

My naive (ha!) Gaussian Naive Bayes classifier is too slow

I am trying to build a film review classifier where I determine if a given review is positive or negative (w/ Python). I'm trying to avoid any other ML libraries so that I can better understand the ...
1
vote
1answer
3k views

How to Combine tfidf with LSTM in keras?

I am classifying emails as spam or ham using LSTM and some of its modified form(by adding constitutional layer at the end). For converting documents into vectors I am using ...
1
vote
1answer
215 views

Assigning a new document to a cluster based on keywords extracted and tf-idf

I have about 40 clusters of documents defined by a combination of k-means clustering algorithm and hand curation. For example, some of the clusters given by k-means are too noisy so they have been ...
0
votes
0answers
18 views

Cosine Similarity: Works with TF-IDF Vectors OR with Probability Vectors?

Using Cosine Similarity is a common method to calculate Semantic Textual Similarity. And it is particularly useful when comparing Sentence Embeddings provided by the Universal Sentence Encoder. ...
0
votes
0answers
12 views

Matching documents from different sets with tfidf and cosine distance

I have two different set of documents S1, S2, with 30 text documents each. Using some text representation method, such as tfidf ...
0
votes
0answers
4 views

Vector dimensionality seems to be implemented incorrectly

I'm trying to implement a fuzzy topic modeling approach in Python based on a paper, which is accommodated with an R implementation from GitHub. In one of the first steps a document term matrix is ...
0
votes
1answer
13 views

How to decide which method to use TFIDF, or BOW

In a huge dataset for NLP it is taking very long time to classify my dataset therefore, trying each feature extraction method separetly is time consuming and not effecient. Is there a way that can ...
0
votes
1answer
57 views

Why does using a standard scalar on my tf idf matrix make it perform better?

I have a tf-idf matrix transformed on a list of tweets from a data set i am using. I have a pipeline where i initiate a StandardScalar and then next have my SVM with a linear kernel and auto gamma as ...
0
votes
0answers
32 views

Is normalizing term weight necessary when cosine similarity is used in retrieval?

When using cosine similarity in information retrieval, document vector length and query vector length are used for normalization. So if TF-IDF is used as a weighting function, then using raw frequency ...
0
votes
0answers
11 views

Can a term weighting function used in text retrieval be compared to one used in text classification?

I came up with a modified version of TF-IDF function for text retrieval task. I want to do retrieval experiments using Vector Space Model and compare my function to some of those proposed in the ...
0
votes
0answers
19 views

The TF-IDF is not matching with the bags of words

I am creating the information gains of all the words present in the vocabulary. However, when I check for feature names of the vectorizer it is different. For bag of words I am using: ...
0
votes
0answers
15 views

Building simple documents search engine

I'm having my first steps in the NLP and at the moment I'm looking forward to building my own documents search engine. I've already got to know with TFIDF in practical way and I've also read about ...
0
votes
0answers
16 views

SKLEARN SGDClassifier prediction accuracy hint?

There is a function predict but is it possible to also hint how much is the predicted category probable? Like prediction of category 1 with 90% confidence, or 2 with 30% confidence etc. Without this I ...
0
votes
1answer
41 views

Is it good practice to remove the numeric values from the text data during preprocessing?

Im doing preprocessing on a text dataset. I have certain numerics in it like: date(1st July) year(2019) tentative values (3-5 years/ 10+ advantages). unique values (room no 31/ user rank 45) ...
0
votes
0answers
27 views

Manually tune tf-idf features in document classification

I am working on a multi-label document classification task with a very small data set (180 labeled documents) and a fairly large number of labels (20). I found that - ignoring label correlations and ...
0
votes
0answers
28 views

TF-IDF Transform duplicating data

I'm working on a Sentiment Analysis task using TF-IDF to build my features and SVC as the classifier. My goal is to make my model to classify the sentiment of all my dataset. I already designed my ...
0
votes
0answers
72 views

TFIDF and TFIDF weighted W2V with Multinomial Naive Bayes?

Although the tfidf vectors don't really follow Multinomial Distribution, yet MultinomialNB works fairly well, why is it so? Also would weighted tfidf w2v work the same way or should I use GaussianNB ...
0
votes
0answers
30 views

How to properly vectorise when I have several text features?

I am having several features as text and I'm using them once for a classification problem and then later for a regression problem. The textual data features themselves aren't categorical. Because as ...
0
votes
0answers
68 views

SKLearn NearestCentroidClassifier score with predict_proba

I'm using the NearestCentroidClassifier combined with TF-IDF for classification of documents. The are linked to a growing number of document groups. I've set sklearns TfIdfVectorizer and the ...
0
votes
1answer
28 views

Solution for TF-IDF Vectorization in Angular project?

While making an Angular project to use my text-classification model on unseen data, i struggle in finding a way how to transform text to TFIDF features. Anyone faced same issue? Maybe a solution on ...
0
votes
1answer
1k views

How to combine nlp and numeric data for a linear regression problem

I'm very new to data science (this is my hello world project), and I have a data set made up of a combination of review text and numerical data such as number of tables. There is also a column for ...
0
votes
2answers
62 views

Is this a good approach to classify tickets which contains description and logs?

I want to classify a dataset of support tickets which mostly contain text in the description field and sometimes server logs in a separate field. The log field is not always there but when it's ...
0
votes
1answer
131 views

KDE on TF-IDF - sensitive bandwidth

I'm clustering texts based on TF-IDF features and DBSCAN (density based), and trying to rank points based on their 'belonging' to the cluster. Since my clustering is density based and my points can ...
-1
votes
1answer
2k views

How to get relevancy score of a term with respect to text/document

I am working on the literature documents. I am able to identify important entities using NER and Ontologies. Now I will like to assign the relevance score to the identified entities with respect to ...