Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
2
votes
4answers
2k views

Format for storing textual data

For an upcoming project, I'm mining textual posts from an online forum, using Scrapy. What is the best way to store this text data? I'm thinking of simply exporting it into a JSON file, but is there a ...
10
votes
1answer
1k views

Multiple labels in supervised learning algorithm

I have a corpus of text with a corresponding topics. For example "A rapper Tupac was shot in LA" and it was labelled as ...
16
votes
2answers
17k views

Extract most informative parts of text from documents

Are there any articles or discussions about extracting part of text that holds the most of information about current document. For example, I have a large corpus of documents from the same domain. ...
12
votes
2answers
332 views

Ethically and Cost-effectively Scaling Data Scrapes

Few things in life give me pleasure like scraping structured and unstructured data from the Internet and making use of it in my models. For instance, the Data Science Toolkit (or ...
5
votes
2answers
553 views

Name Anonymization Software

Although I have seen a few good questions asked about data anonymization, I was wondering if there were answers to this more specific variant. I am seeking a tool (or to design one) that will ...
10
votes
4answers
450 views

Using Clustering in text processing

Hi this is my first question in the Data Science stack. I want to create an algorithm for text classification. Suppose i have a large set of text and articles. Lets say around 5000 plain texts. I ...
5
votes
1answer
71 views

Relation mining of multivariant categorical timeseries without excluding the temporal nature

To all: I have been wracking my brain at this for a while and thought maybe someone here would know of a package or algorithm to handle the following: I have nominal multivariant timeseries that ...
10
votes
3answers
3k views

Log file analysis: extracting information part from value part

I'm trying to build a data set on several log files of one of our products. The different log files have their own layout and own content; I successfully grouped them together, only one step ...
1
vote
0answers
249 views

Analyze paragraphs using Neuroph

Currently we are regularly analyzing sets of paragraphs every month. I would like to automate this and split each paragraphs into chunks of data. To do this I would like to employ a neural network. ...
4
votes
1answer
155 views

Clustering strings inside strings?

I am not sure whether I formulated the question correctly. Basically, what I want to do is: Let's suppose I have a list of 1000 strings which look like this: cvzxcvzxstringcvzcxvz ...
3
votes
1answer
62 views

What circumstances causes two different classifiers to classify data exactly like one another

Okay, here is the background: I am doing text mining, and my basic flow is like this: extract feature (n-gram), reduce feature count, score (tf-idf) and classify. for my own sake i am doing comparison ...
0
votes
1answer
374 views

document clustering by semantic similarity based EMD(earth mover distance)

I want the text-based semantic clustering EMD do. Is there a better way of using LDA to detect topics in text, there are so provide better results? I'm going to do my EMD on discovery topics. Thanks
2
votes
1answer
1k views

Text Classification with mixed features in Random Forests

I am working on a text classification problem on tweets. At the moment I was only considering the content of the tweets as a source of information, and I was using a simple bag of words approach using ...
8
votes
1answer
1k views

Difference between tf-idf and tf with Random Forests

I am working on a text classification problem using Random Forest as classifiers, and a bag-of-words approach. I am using the basic implementation of Random Forests (the one present in scikit), that ...
2
votes
2answers
144 views

General approahces for grouping a continuous variable based on text data?

I have a general methodological question. I have two columns of data, with one a column a numeric variable for age and another column a short character variable for text responses to a question. My ...
2
votes
4answers
1k views

Creating Bag of words

What is the best technology to be used to create my custom bag of words with N-grams to apply to. I want to know a functionality that can be achieved over GUI. I cannot use spot fire as it is not ...
3
votes
2answers
167 views

non query-based document ranking

We have ~500 biomedical documents each of some 1-2 MB. We want to use a non query-based method to rank the documents in order of their unique content score. I'm calling it "unique content" because our ...
3
votes
1answer
3k views

Data sets for evaluating text retrieval quality [closed]

I'm searching for data sets for evaluating text retrieval quality. TF-IDF is a popular similarity measure, but is it the best choice? And which variant is the best choice? Lucenes Scoring for example ...
12
votes
3answers
6k views

Unstructured text classification

I'm going to classify unstructured text documents, namely web sites of unknown structure. The number of classes to which I am classifying is limited (at this point, I believe there is no more than ...
3
votes
1answer
212 views

Query similarity: how much data is used in practice?

I recently read Similarity Measures for Short Segments of Text (Metzler et al.). It describes basic methods for measuring query similarity, and in the paper, the data consists of queries and their ...
17
votes
1answer
12k views

Algorithms for text clustering

I have a problem of clustering huge amount of sentences into groups by their meanings. This is similar to a problem when you have lots of sentences and want to group them by their meanings. What ...
19
votes
3answers
14k views

what is difference between text classification and topic models?

I know the difference between clustering and classification in machine learning, but I don't understand the difference between text classification and topic modeling for documents. Can I use topic ...
4
votes
1answer
423 views

OpenNLP Coreference Resolution (German)

I need to do coreference resolution for German texts and I plan to use OpenNLP to perform this task. As far as I know OpenNLP coreference resolution does not support the German language. Which ...
3
votes
1answer
58 views

Can I classify set of documents using classifying method using limited number of concepts ?

I have set of documents and I want classify them to true and false My question is I have to take the whole words in the documents then I classify them depend on the similarity words in these ...
1
vote
1answer
238 views

SUMMARIST: Automated Text Summarization

There is a text summarization project called SUMMARIST. Apparently it is able to perform abstractive text summarization. I want to give it a try but unfortunately the demo links on the website do not ...
8
votes
1answer
6k views

R error using package tm (text-mining)

I am attempting to use the tm package to convert a vector of text strings to a corpus element. My code looks something like this Corpus(d1$Yes) where d1$Yes is a factor with 124 levels, each ...
3
votes
1answer
877 views

Relative merits of different open source natural language generators

Does anyone know what (from your experience) is the best open source natural language generators (NLG) out there? What are the relative merits of each? I'm looking to do sophisticated text ...
10
votes
3answers
2k views

Unsupervised feature learning for NER

I have implemented NER system with the use of CRF algorithm with my handcrafted features that gave quite good results. The thing is that I used lots of different features including POS tags and lemmas....
5
votes
2answers
153 views

Are there any annotators or Named Entity Recognition for license plate numbers?

Most vehicle license/number plate extractors I've found involve reading a plate from an image (OCR) but I'm interested in something that could tag instances of license plates in a body of text. Are ...
0
votes
1answer
779 views

Commercial Text Summarization Tools [closed]

I'm looking for commercial text summarization tools (APIs, Libraries,...) which are able to perform any of the following tasks: Extractive Multi-Document Summarization (Generic or query-based) ...
34
votes
5answers
10k views

What are some standard ways of computing the distance between documents?

When I say "document", I have in mind web pages like Wikipedia articles and news stories. I prefer answers giving either vanilla lexical distance metrics or state-of-the-art semantic distance metrics,...
27
votes
4answers
24k views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
6
votes
1answer
2k views

What are the main types of NLP annotators?

I'm new to the world of text mining and have been reading up on annotators at places like the UIMA website. I'm encountering many new terms like named entity recognition, tokenizer, lemmatizer, ...
12
votes
2answers
1k views

Preference Matching Algorithm

There's this side project I'm working on where I need to structure a solution to the following problem. I have two groups of people (clients). Group A intends to ...
9
votes
4answers
2k views

Suggest text classifier training datasets

Which freely available datasets can I use to train a text classifier? We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified ...
19
votes
3answers
2k views

How to grow a list of related words based on initial keywords?

I recently saw a cool feature that was once available in Google Sheets: you start by writing a few related keywords in consecutive cells, say: "blue", "green", "yellow", and it automatically generates ...
-4
votes
1answer
515 views

How can I extract news about a particular company from various websites using RODBC package in R? And perform sentiment analysis on the data? [closed]

I would like to extract news about a company from online news by using the RODBC package in R. I would then like to use the extracted data for sentiment analysis. I want to accomplish this in such a ...
19
votes
4answers
3k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...