Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
37 votes
5 answers
11k views

What are some standard ways of computing the distance between documents?

When I say "document", I have in mind web pages like Wikipedia articles and news stories. I prefer answers giving either vanilla lexical distance metrics or state-of-the-art semantic distance metrics,...
user avatar
  • 791
30 votes
4 answers
29k views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
user avatar
30 votes
3 answers
32k views

General approach to extract key text from sentence (nlp)

Given a sentence like: Complimentary gym access for two for the length of stay ($12 value per person per day) What general approach can I take to identify the ...
user avatar
28 votes
3 answers
22k views

What is difference between text classification and topic models?

I know the difference between clustering and classification in machine learning, but I don't understand the difference between text classification and topic modeling for documents. Can I use topic ...
user avatar
  • 331
28 votes
1 answer
16k views

What is Hellinger Distance and when to use it?

I am interested in knowing what really happens in Hellinger Distance (in simple terms). Furthermore, I am also interested in knowing what are types of problems that we can use Hellinger Distance? ...
user avatar
23 votes
3 answers
23k views

Keyword/phrase extraction from Text using Deep Learning libraries

Perhaps this is too broad, but I am looking for references on how to use deep learning in a text summarization task. I have already implemented text summarization using standard word-frequency ...
user avatar
22 votes
5 answers
4k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
user avatar
22 votes
3 answers
5k views

How to grow a list of related words based on initial keywords?

I recently saw a cool feature that was once available in Google Sheets: you start by writing a few related keywords in consecutive cells, say: "blue", "green", "yellow", and it automatically generates ...
user avatar
21 votes
2 answers
17k views

Doc2Vec - How to label the paragraphs (gensim)

I am wondering how to label (tag) sentences / paragraphs / documents with doc2vec in gensim - from a practical standpoint. Do you need to have each sentence / paragraph / document with its own ...
user avatar
  • 692
18 votes
2 answers
23k views

Extract most informative parts of text from documents

Are there any articles or discussions about extracting part of text that holds the most of information about current document. For example, I have a large corpus of documents from the same domain. ...
user avatar
  • 631
18 votes
4 answers
23k views

What is the difference between a hashing vectorizer and a tfidf vectorizer

I'm converting a corpus of text documents into word vectors for each document. I've tried this using a TfidfVectorizer and a HashingVectorizer I understand that a ...
user avatar
  • 735
17 votes
1 answer
12k views

Algorithms for text clustering

I have a problem of clustering huge amount of sentences into groups by their meanings. This is similar to a problem when you have lots of sentences and want to group them by their meanings. What ...
user avatar
16 votes
4 answers
26k views

How to do postal addresses fuzzy matching?

I would like to know how to match postal addresses when their format differ or when one of them is mispelled. So far I've found different solutions but I think that they are quite old and not very ...
user avatar
14 votes
1 answer
280 views

Recognize a grammar in a sequence of fuzzy tokens

I have text documents which contain mainly lists of Items. Each Item is a group of several token from different types: FirstName, LastName, BirthDate, PhoneNumber, City, Occupation, etc. A token is a ...
user avatar
  • 241
13 votes
5 answers
36k views

Clustering with cosine similarity

I have a large data set and a cosine similarity between them. I would like to cluster them using cosine similarity that puts similar objects together without needing to specify beforehand the number ...
user avatar
13 votes
3 answers
7k views

Unstructured text classification

I'm going to classify unstructured text documents, namely web sites of unknown structure. The number of classes to which I am classifying is limited (at this point, I believe there is no more than ...
user avatar
13 votes
3 answers
29k views

What is the difference between NLP and text mining?

As discussed with Sean in this Meta post, I thought it would be nice to have a question which can help people who were confused like me, to know about the differences between text mining and NLP! So, ...
user avatar
  • 8,036
13 votes
3 answers
3k views

Unsupervised feature learning for NER

I have implemented NER system with the use of CRF algorithm with my handcrafted features that gave quite good results. The thing is that I used lots of different features including POS tags and lemmas....
user avatar
  • 631
13 votes
2 answers
362 views

Ethically and Cost-effectively Scaling Data Scrapes

Few things in life give me pleasure like scraping structured and unstructured data from the Internet and making use of it in my models. For instance, the Data Science Toolkit (or ...
user avatar
  • 1,879
13 votes
4 answers
11k views

Alternatives to TF-IDF and Cosine Similarity when comparing documents of differing formats

I've been working on a small, personal project which takes a user's job skills and suggests the most ideal career for them based on those skills. I use a database of job listings to achieve this. At ...
user avatar
12 votes
2 answers
2k views

Preference Matching Algorithm

There's this side project I'm working on where I need to structure a solution to the following problem. I have two groups of people (clients). Group A intends to ...
user avatar
  • 141
12 votes
2 answers
7k views

Document classification using convolutional neural network

I'm trying to use CNN (convolutional neural network) to classify documents. CNN for short text/sentences has been studied in many papers. However, it seems that no papers have used CNN for long text ...
user avatar
11 votes
1 answer
3k views

How to determine if character sequence is English word or noise

What kind of features you will try to extract from list of words for future predicting, is it existing word or just mess of characters ? There is description of task that I found there. You have to ...
user avatar
11 votes
4 answers
491 views

Using Clustering in text processing

Hi this is my first question in the Data Science stack. I want to create an algorithm for text classification. Suppose i have a large set of text and articles. Lets say around 5000 plain texts. I ...
user avatar
  • 213
11 votes
1 answer
3k views

applying word2vec on small text files

I'm totally new to word2vec so pls bear it with me. I have a set of text files each containing a set of tweets, between 1000-3000. I have chosen a common keyword ("kw1") and wants to find semantically ...
user avatar
  • 211
10 votes
3 answers
19k views

Public dataset for news articles with their associated categories

I am wondering if there are any public datasets of Google news with various news categories such as politics, entertainment, lifestyle, general news, sports etc. I want to use such dataset for topic ...
user avatar
  • 213
10 votes
1 answer
1k views

How to determine the complexity of an English sentence?

I am working on an app to help people learn English as a second language. I have validated that sentences help in learning a language by providing extra context. I did that by conducting a small ...
user avatar
10 votes
3 answers
1k views

Vector space model cosine tf-idf for finding similar documents

Have corpus of over million documents For a given document want to find similar documents using cosine as in vector space model $d_1 \cdot d_2 / ( ||d_1|| ||d_2|| )$ All tf have been ...
user avatar
  • 188
10 votes
1 answer
1k views

Multiple labels in supervised learning algorithm

I have a corpus of text with a corresponding topics. For example "A rapper Tupac was shot in LA" and it was labelled as ...
user avatar
10 votes
3 answers
3k views

Log file analysis: extracting information part from value part

I'm trying to build a data set on several log files of one of our products. The different log files have their own layout and own content; I successfully grouped them together, only one step ...
user avatar
10 votes
1 answer
2k views

Text-Classification-Problem: Is Word2Vec/NN the best approach?

I am looking to design a system that given a paragraph of text will be able to categorize it and identify the context: Is trained with user generated text paragraphs (like comments/questions/answers) ...
user avatar
  • 101
10 votes
1 answer
5k views

How much training data does Word2Vec need?

I'd like to compare the difference among the same word mentioned in different sources. That is, how authors differ in their usage of ill-defined words, such as "democracy". A brief plan was Take the ...
user avatar
9 votes
4 answers
3k views

Suggest text classifier training datasets

Which freely available datasets can I use to train a text classifier? We are trying to enhance our users engagement by recommending the most related content for him, so we thought If we classified ...
user avatar
9 votes
4 answers
3k views

How to deal with spelling errors NLP

I have some data where the main column is the description of one product. The main task is to extract the name of some product from this column, where it sometimes is spelled wrong and amended in ...
user avatar
  • 221
9 votes
1 answer
2k views

Difference between tf-idf and tf with Random Forests

I am working on a text classification problem using Random Forest as classifiers, and a bag-of-words approach. I am using the basic implementation of Random Forests (the one present in scikit), that ...
user avatar
  • 565
9 votes
2 answers
819 views

what machine/deep learning/ nlp techniques are used to classify a given words as name, mobile number, address, email, state, county, city etc

I am trying to generate an intelligent model which can scan a set of words or strings and classify them as names, mobile numbers, addresses, cities, states, countries and other entities using machine ...
user avatar
8 votes
4 answers
4k views

How to learn spam email detection?

I want to learn how a spam email detector is done. I'm not trying to build a commercial product, it'll be a serious learning exercise for me. Therefore, I'm looking for resources, such as existing ...
user avatar
  • 3,390
8 votes
1 answer
23k views

Resume Parsing - extracting skills from resume using Machine Learning

I am trying to extract a skill set of an employee from his/her resume. I have resumes stored as plain text in Database. I do not have predefined skills in this case. How should I approach this problem?...
user avatar
  • 1,133
8 votes
3 answers
3k views

How evaluate text clustering?

What metrics can be used for evaluating text clustering models? I used tf-idf + k-means, ...
user avatar
8 votes
1 answer
3k views

Which classification algorithms to try for classifying text data into 300 categories

I have 40000 rows of text data of health care domain. Data has one column for text (2-5 sentences) and one column for its category. I want to classify that into 300 categories. Some categories are ...
user avatar
8 votes
1 answer
18k views

Text extraction from documents using NLP or Deep Learning

I am looking for references(Papers/github projects) on how to use deep learning in a text extraction task. Recently I was given a task to extract important information from documents of similar type, ...
user avatar
8 votes
1 answer
16k views

How to extract paragraphs from text document?

I have extracted text data from pdf files of annual reports of companies using pdftotext. The extracted file content looks like: Sample pdf file is here FORWARD-LOOKING STATEMENTS In this Annual ...
user avatar
  • 181
8 votes
4 answers
885 views

Classifying Email in R

I'm working on a project in R where I have roughly 1200 emails from a company, most of which are labeled class$_{1}$ or class$_{2}$, which are the types of requests. Roughly 1000 emails are labeled ...
user avatar
7 votes
1 answer
2k views

What are the main types of NLP annotators?

I'm new to the world of text mining and have been reading up on annotators at places like the UIMA website. I'm encountering many new terms like named entity recognition, tokenizer, lemmatizer, ...
user avatar
7 votes
2 answers
8k views

Date Extraction in Python

I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes: 2019-02-20 20.02.2019 ("German format") 02/2019 ("February 2019") "...
user avatar
  • 17.6k
7 votes
4 answers
8k views

How to give name to topics created using LDA?

I have categorized 800,000 documents into 500 categories using the Mahout topic modelling. Instead of representing the topic using the top 5/10 words for each topics, I want to infer a generic name ...
user avatar
  • 71
7 votes
1 answer
8k views

R error using package tm (text-mining)

I am attempting to use the tm package to convert a vector of text strings to a corpus element. My code looks something like this Corpus(d1$Yes) where ...
user avatar
  • 79
7 votes
3 answers
224 views

How to plot clusters in nice a way?

I have a large text dataset clusterized. Each cluster is represented by a centroid of the vectorized texts that belong to it, the number of texts, the created date, and other parameters. I can't plot ...
user avatar
7 votes
1 answer
157 views

which deep learning text classifier is good for health data

I have a data set like this: ...
user avatar
  • 321
6 votes
3 answers
233 views

What is an alternative name for "Unstructured Data"?

I'm writing my thesis at the moment, and for some time - due to a lack of a proper alternative - I've stuck with "unstructured data" for referring to natural, free flowing text, e.g. Wikipedia ...
user avatar

1
2 3 4 5
12