Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
35
votes
5answers
10k views

What are some standard ways of computing the distance between documents?

When I say "document", I have in mind web pages like Wikipedia articles and news stories. I prefer answers giving either vanilla lexical distance metrics or state-of-the-art semantic distance metrics,...
28
votes
4answers
26k views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
27
votes
3answers
31k views

General approach to extract key text from sentence (nlp)

Given a sentence like: Complimentary gym access for two for the length of stay ($12 value per person per day) What general approach can I take to identify the ...
22
votes
3answers
16k views

what is difference between text classification and topic models?

I know the difference between clustering and classification in machine learning, but I don't understand the difference between text classification and topic modeling for documents. Can I use topic ...
20
votes
3answers
20k views

Keyword/phrase extraction from Text using Deep Learning libraries

Perhaps this is too broad, but I am looking for references on how to use deep learning in a text summarization task. I have already implemented text summarization using standard word-frequency ...
20
votes
1answer
8k views

What is Hellinger Distance and when to use it?

I am interested in knowing what really happens in Hellinger Distance (in simple terms). Furthermore, I am also interested in knowing what are types of problems that we can use Hellinger Distance? ...
19
votes
3answers
3k views

How to grow a list of related words based on initial keywords?

I recently saw a cool feature that was once available in Google Sheets: you start by writing a few related keywords in consecutive cells, say: "blue", "green", "yellow", and it automatically generates ...
19
votes
2answers
14k views

Doc2Vec - How to label the paragraphs (gensim)

I am wondering how to label (tag) sentences / paragraphs / documents with doc2vec in gensim - from a practical standpoint. Do you need to have each sentence / paragraph / document with its own ...
18
votes
4answers
3k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
17
votes
1answer
12k views

Algorithms for text clustering

I have a problem of clustering huge amount of sentences into groups by their meanings. This is similar to a problem when you have lots of sentences and want to group them by their meanings. What ...
16
votes
2answers
21k views

Extract most informative parts of text from documents

Are there any articles or discussions about extracting part of text that holds the most of information about current document. For example, I have a large corpus of documents from the same domain. ...
14
votes
4answers
19k views

How to do postal addresses fuzzy matching?

I would like to know how to match postal addresses when their format differ or when one of them is mispelled. So far I've found different solutions but I think that they are quite old and not very ...
13
votes
3answers
16k views

What is the difference between a hashing vectorizer and a tfidf vectorizer

I'm converting a corpus of text documents into word vectors for each document. I've tried this using a TfidfVectorizer and a HashingVectorizer I understand that a ...
13
votes
2answers
356 views

Ethically and Cost-effectively Scaling Data Scrapes

Few things in life give me pleasure like scraping structured and unstructured data from the Internet and making use of it in my models. For instance, the Data Science Toolkit (or ...
13
votes
4answers
6k views

Alternatives to TF-IDF and Cosine Similarity when comparing documents of differing formats

I've been working on a small, personal project which takes a user's job skills and suggests the most ideal career for them based on those skills. I use a database of job listings to achieve this. At ...
13
votes
1answer
260 views

Recognize a grammar in a sequence of fuzzy tokens

I have text documents which contain mainly lists of Items. Each Item is a group of several token from different types: FirstName, LastName, BirthDate, PhoneNumber, City, Occupation, etc. A token is a ...
12
votes
3answers
6k views

Unstructured text classification

I'm going to classify unstructured text documents, namely web sites of unknown structure. The number of classes to which I am classifying is limited (at this point, I believe there is no more than ...
12
votes
2answers
1k views

Preference Matching Algorithm

There's this side project I'm working on where I need to structure a solution to the following problem. I have two groups of people (clients). Group A intends to ...
11
votes
1answer
2k views

How to determine if character sequence is English word or noise

What kind of features you will try to extract from list of words for future predicting, is it existing word or just mess of characters ? There is description of task that I found there. You have to ...
11
votes
4answers
477 views

Using Clustering in text processing

Hi this is my first question in the Data Science stack. I want to create an algorithm for text classification. Suppose i have a large set of text and articles. Lets say around 5000 plain texts. I ...
11
votes
3answers
3k views

Unsupervised feature learning for NER

I have implemented NER system with the use of CRF algorithm with my handcrafted features that gave quite good results. The thing is that I used lots of different features including POS tags and lemmas....
11
votes
2answers
6k views

Document classification using convolutional neural network

I'm trying to use CNN (convolutional neural network) to classify documents. CNN for short text/sentences has been studied in many papers. However, it seems that no papers have used CNN for long text ...
11
votes
1answer
3k views

applying word2vec on small text files

I'm totally new to word2vec so pls bear it with me. I have a set of text files each containing a set of tweets, between 1000-3000. I have chosen a common keyword ("kw1") and wants to find semantically ...
10
votes
1answer
869 views

How to determine the complexity of an English sentence?

I am working on an app to help people learn English as a second language. I have validated that sentences help in learning a language by providing extra context. I did that by conducting a small ...
10
votes
3answers
1k views

Vector space model cosine tf-idf for finding similar documents

Have corpus of over million documents For a given document want to find similar documents using cosine as in vector space model $d_1 \cdot d_2 / ( ||d_1|| ||d_2|| )$ All tf have been ...
10
votes
1answer
1k views

Multiple labels in supervised learning algorithm

I have a corpus of text with a corresponding topics. For example "A rapper Tupac was shot in LA" and it was labelled as ...
10
votes
3answers
3k views

Log file analysis: extracting information part from value part

I'm trying to build a data set on several log files of one of our products. The different log files have their own layout and own content; I successfully grouped them together, only one step ...
10
votes
1answer
2k views

Text-Classification-Problem: Is Word2Vec/NN the best approach?

I am looking to design a system that given a paragraph of text will be able to categorize it and identify the context: Is trained with user generated text paragraphs (like comments/questions/answers) ...
10
votes
1answer
4k views

How much training data does word2vec need?

I'd like to compare the difference among the same word mentioned in different sources. That is, how authors differ in their usage of ill-defined words, such as "democracy". A brief plan was Take the ...
9
votes
4answers
3k views

Suggest text classifier training datasets

Which freely available datasets can I use to train a text classifier? We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified ...
9
votes
2answers
369 views

what machine/deep learning/ nlp techniques are used to classify a given words as name, mobile number, address, email, state, county, city etc

I am trying to generate an intelligent model which can scan a set of words or strings and classify them as names, mobile numbers, addresses, cities, states, countries and other entities using machine ...
8
votes
3answers
20k views

What is the difference between NLP and text mining?

As discussed with Sean in this Meta post, I thought it would be nice to have a question which can help people who were confused like me, to know about the differences between text mining and NLP! So, ...
8
votes
5answers
21k views

Clustering with cosine similarity

I have a large data set and a cosine similarity between them. I would like to cluster them using cosine similarity that puts similar objects together without needing to specify beforehand the number ...
8
votes
4answers
4k views

How to learn spam email detection?

I want to learn how a spam email detector is done. I'm not trying to build a commercial product, it'll be a serious learning exercise for me. Therefore, I'm looking for resources, such as existing ...
8
votes
1answer
7k views

R error using package tm (text-mining)

I am attempting to use the tm package to convert a vector of text strings to a corpus element. My code looks something like this Corpus(d1$Yes) where ...
8
votes
3answers
2k views

How evaluate text clustering?

What metrics can be used for evaluating text clustering models? I used tf-idf + k-means, ...
8
votes
1answer
3k views

Which classification algorithms to try for classifying text data into 300 categories

I have 40000 rows of text data of health care domain. Data has one column for text (2-5 sentences) and one column for its category. I want to classify that into 300 categories. Some categories are ...
8
votes
1answer
2k views

Difference between tf-idf and tf with Random Forests

I am working on a text classification problem using Random Forest as classifiers, and a bag-of-words approach. I am using the basic implementation of Random Forests (the one present in scikit), that ...
8
votes
4answers
744 views

Classifying Email in R

I'm working on a project in R where I have roughly 1200 emails from a company, most of which are labeled class$_{1}$ or class$_{2}$, which are the types of requests. Roughly 1000 emails are labeled ...
7
votes
1answer
2k views

What are the main types of NLP annotators?

I'm new to the world of text mining and have been reading up on annotators at places like the UIMA website. I'm encountering many new terms like named entity recognition, tokenizer, lemmatizer, ...
7
votes
1answer
18k views

Resume Parsing - extracting skills from resume using Machine Learning

I am trying to extract a skill set of an employee from his/her resume. I have resumes stored as plain text in Database. I do not have predefined skills in this case. How should I approach this problem?...
7
votes
3answers
13k views

Public dataset for news articles with their associated categories

I am wondering if there are any public datasets of Google news with various news categories such as politics, entertainment, lifestyle, general news, sports etc. I want to use such dataset for topic ...
7
votes
2answers
3k views

Date Extraction in Python

I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes: 2019-02-20 20.02.2019 ("German format") 02/2019 ("February 2019") "...
7
votes
3answers
156 views

How to plot clusters in nice a way?

I have a large text dataset clusterized. Each cluster is represented by a centroid of the vectorized texts that belong to it, the number of texts, the created date, and other parameters. I can't plot ...
7
votes
1answer
14k views

Text extraction from documents using NLP or Deep Learning

I am looking for references(Papers/github projects) on how to use deep learning in a text extraction task. Recently I was given a task to extract important information from documents of similar type, ...
7
votes
1answer
138 views

which deep learning text classifier is good for health data

I have a data set like this: ...
7
votes
1answer
12k views

How to extract paragraphs from text document?

I have extracted text data from pdf files of annual reports of companies using pdftotext. The extracted file content looks like: Sample pdf file is present at http://www.heromotocorp.com/en-in/uploads/...
6
votes
4answers
861 views

How to deal with spelling errors NLP

I have some data where the main column is the description of one product. The main task is to extract the name of some product from this column, where it sometimes is spelled wrong and amended in ...
6
votes
3answers
187 views

What is an alternative name for “Unstructured Data”?

I'm writing my thesis at the moment, and for some time - due to a lack of a proper alternative - I've stuck with "unstructured data" for referring to natural, free flowing text, e.g. Wikipedia ...
6
votes
2answers
1k views

Classifying survey response text SVM

I have 800 responses to an open-ended survey question. Each response is categorized into 3 categories based on a list of 70 categories. These categories are things like "stronger leadership", "better ...

1
2 3 4 5
11