Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
1answer
355 views

Automatic question categorization when we know important words in each category

I am currently working on a question categorization problem where I automatically want to assign a category to the question. The question set I have is unlabelled. The categories for the problem are ...
-1
votes
1answer
94 views

Extracting paragraph from a document based on Numerals

I am new to the Data Science. The problem I want to solve is relatively simple in terms of the problem statement. Given a Numbered document (usually a pdf) with/without an index, I need to extract ...
0
votes
1answer
922 views

Regex remove 1-2 character sequences: hyphens literal, not word boundaries [closed]

I need a regex in R to exclude 1 or 2-character words, but which does not treat hyphens as word boundaries. Here is an example: ...
1
vote
0answers
529 views

Perplexity increasing on Test DataSet in LDA (Topic Modelling)

I was plotting the perplexity values on LDA models (R) by varying topic numbers. Already train and test corpus was created. Unfortunately, perplexity is increasing with increased number of topics on ...
1
vote
1answer
668 views

How can I find contextually related words and classify into custom tags/labels?

PROBLEM: Suppose if I have a small dataset containing some words and their tags/labels. The main task is to provide tags to other words(which are not in the dataset) based on their contextual ...
2
votes
1answer
70 views

Text annotating process, quality vs quantity?

I have a question regarding annotating text data for classification. Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time ...
0
votes
1answer
30 views

Text annotating process, quality vs quantity? [duplicate]

I have a question regarding annotating text data for classification. Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time ...
-3
votes
1answer
33 views

Algorithms/services to know an “iPhone case” is not an “iPhone”, in the context of complex item descriptions? [closed]

We are trying to implement a highly accurate search, based on user-entered search terms, into a large product database. For example, if the user searches for "iPhone", then one of these is ...
0
votes
1answer
83 views

How to go about text mining for suggestions/Tips in reviews for restaurants? [closed]

For example for restaurants reviews usually have suggestions like "Go in the evenings", "order the so and so sauce with this dish" or even "TIP: ask for the blah blah blah" How can I detect such ...
0
votes
1answer
372 views

Can we apply community detection algorithms for word vector space?

As I understand we can apply community detection algorithms such as Louvain to detect communities in a social network (i.e. involves people). But I am quite interested in knowing if we can use the ...
4
votes
2answers
882 views

Compare two topic modelling sets

I have two sets of newspaper articles where I train the first newspaper dataset separately to get the topics per each newspaper article. ...
4
votes
1answer
1k views

What are real world applications of Doc2Vec?

I am new to Doc2Vec. As I understand Doc2Vec group similar documents based on the context of their words. I have a set of newspaper documents and I want to identify what are the main topics of the ...
0
votes
2answers
621 views

Classification of obfuscated text data

I am relatively new to data science and have an exercise task. This consists of the classifications of excerpts of texts. However, the texts are obfuscated such that one cannot read the words, spaces ...
1
vote
0answers
469 views

TF-IDF Augmented Frequency vs Cosine Normalization

I am using TF-IDF for text classification and have been curious about the following two concepts. The augmented term frequency which is basically used for weighting in order to eliminate the bias ...
0
votes
1answer
135 views

Extracting specific information from multiple unstructured website [closed]

I want to extract specific information e.g the names of all professor teaching subject xyz from multiple college faculty directory. Each website has different structure and the content is not uniform (...
4
votes
3answers
9k views

Public dataset for news articles with their associated categories

I am wondering if there are any public datasets of Google news with various news categories such as politics, entertainment, lifestyle, general news, sports etc. I want to use such dataset for topic ...
4
votes
1answer
861 views

Group similar words under one topic and assign them a title

I am working on a natural language processing data problem and I have selected some keywords from it as features. I want to group them under one heading. But I can't find any method or algorithm to do ...
2
votes
1answer
246 views

Feature extraction from the text

I am a newbie in machine learning but I have a coursework to create program that can extract some concrete features from the given text. For example: If I want to extract number of red apples and ...
5
votes
2answers
875 views

What algorithm can help me discover synonyms?

In the context of text mining, I'd like to discover potential synonyms in my dataset. The current dataset is stackexchange's stackoverflow data on archive.org. The result doesn't have to be perfect, ...
1
vote
1answer
356 views

Tokenize text with both American and English words

I need to tokenize a corpus of abstracts from an international conference. The abstracts are usually American English but sometimes British English. Consequently, I get 2 tokens for “organization” ...
0
votes
1answer
137 views

What is required in Affinity Propagation

I am hoping to use affinity propagation to cluster my data using sklearn. But I came across a question whether to use a distance matrix or similarity matrix in the <...
0
votes
1answer
640 views

Most meaningful words from TF-IDF

I have a TF-IDF representation of labeled documents (say 5 labels). Is there a way to estimate the top-n words that are most likely to contribute to the classification, before applying any model (...
1
vote
2answers
384 views

Eliminate low quality predictions in a classification task

Here is some background on the problem. My aim is to classify text into some categories. I would like to get only good quality predictions from the model. If the model is not confident, I would like ...
1
vote
1answer
433 views

Limits of Hellinger distance values

I am calculating Hellinger distance for different vectors. I initially assumed that the value returned by it in in the range of 0 to 1. However for the following two vectors I received Hellinger ...
2
votes
1answer
1k views

How to extract titles from documents?

How can one automate the extraction of a relevant title from a given document (docx, pdf, etc..)? Some thoughts: Intuitively likely to be the first line in the text Cannot be something like a date, "...
1
vote
1answer
457 views

Choosing the right corpus to build a TF IDF Vectorizer used for comparing the similarity of two strings

I am working on evaluating when a pair of string objects can be considered equal (e.g. given that we are talking about journals, is "international journal of air and water pollution" the same of "air ...
8
votes
5answers
14k views

Clustering with cosine similarity

I have a large data set and a cosine similarity between them. I would like to cluster them using cosine similarity that puts similar objects together without needing to specify beforehand the number ...
1
vote
0answers
219 views

Hellinger Distance in Gensim

I have set of documents as follows where each document has set of words that represents the content of it. ...
19
votes
1answer
5k views

What is Hellinger Distance and when to use it?

I am interested in knowing what really happens in Hellinger Distance (in simple terms). Furthermore, I am also interested in knowing what are types of problems that we can use Hellinger Distance? ...
-1
votes
1answer
69 views

Text standardisation for manually entered data

I am working on a project that involves dealing with manually entered text data. I have a dataset of customs records where the customs officers manually enter the name and address of companies ...
4
votes
1answer
135 views

Logic in sentence : tree representation

I have sentences telling me to who a shop is opened to: "cats, dogs or birds" (1) "young dogs with collar" (2) "old cats or yellow birds" (3) etc... I would like to design an algorithm that will ...
2
votes
2answers
263 views

Detect related sentences

This question is related to "How to grow a list of related words based on initial keywords?" In the previous question they attempt to get similar words to a given word. However, I am interested in ...
2
votes
0answers
590 views

Is there an algorithm or NN to match two documents, basically not closely similar?

Is there an algorithm or NN to match two documents? One is a claim description (e.g. a CV or product offer) and another is a requirements description (e.g. vacancy description or RFP). They are not ...
0
votes
1answer
802 views

Extract and normalize date from string for the proper text mining

I am trying to perform string delivery classification (X: delivery_string, y: delivery_string_relevance (values 0 or 1)). I am using DTM (document term matrix) for the feature extraction, and ...
-1
votes
2answers
866 views

Best way to tokenize tweet

While working with Twitter datasets, one thing that always confuses me is, How to tokenize the tweets. I have seen different open-source implementations using different schemes for tokenization. ...
10
votes
2answers
11k views

What is the difference between a hashing vectorizer and a tfidf vectorizer

I'm converting a corpus of text documents into word vectors for each document. I've tried this using a TfidfVectorizer and a HashingVectorizer I understand that a ...
2
votes
1answer
268 views

using LSTM encoder decoder framework to generate sentences

i am trying out multiple approaches to generate domain specific data (sentences).For e.g. if i am building a data set to train a chatbot in the travel domain, i would like to take pairs of sentences ...
2
votes
1answer
2k views

Apply SVM on LDA in python

hope someone kindly put time here, my approach is like this: TFIDF -> LDA -> SVM I am using LDA to extract topics. I want to do topic modelling and use the topics as features to do document ...
-1
votes
1answer
100 views

Categorize observations with inconsistent text descriptions

Given data table with inconsistent item descriptions, how could I most effectively assign an item category using R (i.e. dplyr), MySQL, or Python? An R based ...
0
votes
1answer
70 views

Dataset - Sample pdfs for text processing?

I'm looking for a rather large amount of pdf files for testing my text processing program. Tried looking for an open site to get like some thousand pdfs, but wasn't able to find anything. I don't ...
1
vote
1answer
3k views

finding themes from text documents

I have a text documents that contain 1000s of abstracts from medical whitepapers. I want to find themes from that text. Any suggestions other than text clustering since clustering helped me to find ...
0
votes
1answer
264 views

How can i build a machine learning model to identify a specific word? [closed]

I am trying to build a model that can be used to identify the word sales whenever a group of text are passed through the model. Should I use Azure or Python for this? I have done the text ...
1
vote
1answer
304 views

Word2vec using gensim

I am using gensim library to find most similar words to some words that I have. Using 10000 data samples (short text mainly 1-2 sentences) to train, I get really bad results! Why this is so? Also by ...
1
vote
1answer
76 views

Excel file merge with different headers but same data

I need to merge data from 1000s of excel files provided by different operations managers on productivity and other reports. The excel files have similarity of data but the headers are all custom since ...
1
vote
2answers
217 views

How to Classify Documents whether they are similar to previous documents or not?

I am a newbie in machine learning.I have 100 text documents.I need to build a model on those 100 text documents and if i give new documents it has to give whether this new document is similar to those ...
2
votes
1answer
2k views

How to compare LDA and TF-IDF?

I am doing text mining to extract topics from documents. I started with Latent Dirichlet Allocation (LDA), which worked great, but then I came across TF-IDF with K-Means clustering, which worked ...
1
vote
1answer
100 views

Extracting Part of Speech (Source and Destinations) using text mining/NLP?

I need to extract the source and destination terms from the text documents using text mining / NLP / Information Retrieval ? Example input: I am travelling from New York to London. I am ...
1
vote
1answer
158 views

Prime model with mock data, good idea or not?

I am currently working on text classifier with some pretty unique characteristics. The data is composed of about 2K categories but 98% of the data lives in just one of those 2K categories. However, ...
0
votes
1answer
299 views

Combining binary features with textual ones

I have a binary feature that i want to use it with textual features i.e. unigrams. I use logistic regression and TF/IDF for representing text. So i simply add a unique feature, say ss or oo, to text ...
9
votes
1answer
596 views

How to determine the complexity of an English sentence?

I am working on an app to help people learn English as a second language. I have validated that sentences help in learning a language by providing extra context. I did that by conducting a small ...