Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
4
votes
3answers
3k views

How to give name to topics created using LDA?

I have categorized 800,000 documents into 500 categories using the Mahout topic modelling. Instead of representing the topic using the top 5/10 words for each topics, I want to infer a generic name ...
17
votes
2answers
13k views

Doc2Vec - How to label the paragraphs (gensim)

I am wondering how to label (tag) sentences / paragraphs / documents with doc2vec in gensim - from a practical standpoint. Do you need to have each sentence / paragraph / document with its own ...
3
votes
1answer
4k views

What are useful evaluation metrics used in machine learning

I am using CNN in order to predict codes after analyzing text. As an example, I will write "I am crazy" .. the model will predict some code " X321". All this based on CNN. I want to evaluate my ...
16
votes
2answers
18k views

Extract most informative parts of text from documents

Are there any articles or discussions about extracting part of text that holds the most of information about current document. For example, I have a large corpus of documents from the same domain. ...
12
votes
2answers
1k views

Preference Matching Algorithm

There's this side project I'm working on where I need to structure a solution to the following problem. I have two groups of people (clients). Group A intends to ...
5
votes
2answers
998 views

Classifying survey response text SVM

I have 800 responses to an open-ended survey question. Each response is categorized into 3 categories based on a list of 70 categories. These categories are things like "stronger leadership", "better ...
25
votes
3answers
29k views

General approach to extract key text from sentence (nlp)

Given a sentence like: Complimentary gym access for two for the length of stay ($12 value per person per day) What general approach can I take to identify the ...
27
votes
4answers
24k views

What algorithms should I use to perform job classification based on resume data?

Note that I am doing everything in R. The problem goes as follow: Basically, I have a list of resumes (CVs). Some candidates will have work experience before and some don't. The goal here is to: ...
19
votes
3answers
18k views

Keyword/phrase extraction from Text using Deep Learning libraries

Perhaps this is too broad, but I am looking for references on how to use deep learning in a text summarization task. I have already implemented text summarization using standard word-frequency ...
19
votes
1answer
5k views

What is Hellinger Distance and when to use it?

I am interested in knowing what really happens in Hellinger Distance (in simple terms). Furthermore, I am also interested in knowing what are types of problems that we can use Hellinger Distance? ...
8
votes
5answers
16k views

Clustering with cosine similarity

I have a large data set and a cosine similarity between them. I would like to cluster them using cosine similarity that puts similar objects together without needing to specify beforehand the number ...
19
votes
3answers
2k views

How to grow a list of related words based on initial keywords?

I recently saw a cool feature that was once available in Google Sheets: you start by writing a few related keywords in consecutive cells, say: "blue", "green", "yellow", and it automatically generates ...
12
votes
4answers
5k views

Alternatives to TF-IDF and Cosine Similarity when comparing documents of differing formats

I've been working on a small, personal project which takes a user's job skills and suggests the most ideal career for them based on those skills. I use a database of job listings to achieve this. At ...
19
votes
4answers
3k views

How to annotate text documents with meta-data?

Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document: ...
11
votes
1answer
2k views

applying word2vec on small text files

I'm totally new to word2vec so pls bear it with me. I have a set of text files each containing a set of tweets, between 1000-3000. I have chosen a common keyword ("kw1") and wants to find semantically ...
9
votes
1answer
658 views

How to determine the complexity of an English sentence?

I am working on an app to help people learn English as a second language. I have validated that sentences help in learning a language by providing extra context. I did that by conducting a small ...
14
votes
4answers
14k views

How to do postal addresses fuzzy matching?

I would like to know how to match postal addresses when their format differ or when one of them is mispelled. So far I've found different solutions but I think that they are quite old and not very ...
4
votes
2answers
1k views

Date Extraction in Python

I would like to extract all date information from a given document. Essentially, I guess this can be done with a lot of regexes: 2019-02-20 20.02.2019 ("German format") 02/2019 ("February 2019") "...
9
votes
1answer
4k views

How much training data does word2vec need?

I'd like to compare the difference among the same word mentioned in different sources. That is, how authors differ in their usage of ill-defined words, such as "democracy". A brief plan was Take the ...
7
votes
1answer
503 views

Comparing two Corpora using Topic Model

I want to compare two corpora (two different collections of texts) using Topic Modeling. I trained the model separately on the two collections and manually matched similar topics based on their ...
4
votes
1answer
425 views

OpenNLP Coreference Resolution (German)

I need to do coreference resolution for German texts and I plan to use OpenNLP to perform this task. As far as I know OpenNLP coreference resolution does not support the German language. Which ...
3
votes
1answer
2k views

Text Mining on Large Dataset

I have a large data set(460 Mb) which has a column - Log with 386551 rows. I wish to use clustering and N-Gram approach to form word cloud. My code is as follows: ...
5
votes
2answers
569 views

Clustering Observations by String Sequences (Python/Pandas df)

I have a dataset consisting of approximately 2 million unique observations. It was initially a set of ID's and URLs. The goal is to cluster the ID's based on the URLs looked at. I transformed both ...
4
votes
2answers
174 views

How would you categorize email subjects to find similar emails?

I have a list of email subjects like: <XYZ> commented on <ABC> Weekly review for <Company> Your account is ready And I want to find patterns ...
3
votes
3answers
6k views

Appropriate algorithm for string (not document) classification?

I am trying to classify a large-ish number of small strings (millions) into about 10 disjunct categories. Examples of classes and strings for each class include: ...
2
votes
2answers
274 views

Detect related sentences

This question is related to "How to grow a list of related words based on initial keywords?" In the previous question they attempt to get similar words to a given word. However, I am interested in ...
2
votes
1answer
74 views

Text annotating process, quality vs quantity?

I have a question regarding annotating text data for classification. Assume we have ten volunteers who are about to annotate a large number of texts into label A or B. They probably won't have time ...
1
vote
1answer
192 views

What is NLP technique to generalize manually created rules in text?

Let's say we have a free text containing key-value entities. Example: "... patient's tumour has width 6 cm and height 5 cm" Then an expert comes, marks it as important, thus we do have the rule for ...
0
votes
1answer
2k views

Algorithm for classification of words into given categories [closed]

I'm working with textual data from medical field. I have a list of words and I want to build an algorithm that can classify each word into one or more given categories, like Medicine_Name ...
0
votes
1answer
22 views

Finding cosine similarity score

I have a dataframe that looks like this: sentence intent hi greeting hello greeting buy this buy whats up conversation . . What I'd like ...
0
votes
1answer
162 views

Text classification and clustering with complete date imbalance

I have a set of scientific papers of authors who have common research interests from PUBMED and I would like to: Clustering papers and extracting features from them in order to find other authors ...