Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it ...

0
votes
1answer
5 views

How to auto tag texts

Suppose we have predefined list of tags Tag #1, Tag #2, ..., Tag #N and we want to assign ...
1
vote
0answers
8 views

Learning to parse elements from text and assigning them to categories

I am trying to generate a table with values parsed from unstructured text. Below are a couple of examples of possibly thousands of entries. For each entry, I would like to identify the title, assign ...
-1
votes
0answers
7 views

what is the technique to identify whether a text is a command or not with natural language processing?

We need to filter commands such as "Shut up" from utterances (which are in textual form). What is the technique to use for such tasks.
0
votes
0answers
10 views

How to extract pre-defined knowledge points from texts? [closed]

The challenge before me is to extract data points from short unstructured text. The example I have is of a large group chat for car mechanics in a city. On the chat, mechanics can ask eachother for ...
0
votes
0answers
10 views

Selecting data in LDAvis Rshinny app

Hi please I have this R shiny code which I would like help on how to modify it server.R library("LDAvis") library(NLP) library(tm) library("servr") library(shiny) library("lda") Load dataframe ...
1
vote
0answers
10 views

How to give names/labels to topics in LDA

I want to give labels to different topics created using LDA. I don't want to do it manually. I saw some papers on automatic labeling but I am still confused. How can I use the information produced ...
1
vote
1answer
24 views

Resume Parsing - extracting skills from resume using Machine Learning

I am trying to extract a skill set of an employee from his/her resume. I have resumes stored as plain text in Database. I do not have predefined skills in this case. How should I approach this problem?...
0
votes
0answers
10 views

How to extract event from news article and merge similar event together

TL;DR Is there a way to extract events from news and cluster similar news event together? I am following the ACE methodology currently, is that a viable way? Longer version: I am currently working ...
0
votes
1answer
20 views

How to interpret Hashingvectorizer representation?

I cannot really understand the logic behind Hashingvectorizer for text feature extraction. I can follow the logic of Bag of Word or TFiDF where the features are values for all/certain words/N-grams ...
0
votes
1answer
46 views

Text Similarities: which nlp methods to use?

I have data where there is text for each user A visiting a business B. I want to find similarity between each user using their text. Question 1: Which NLP method should I start with? I have tried ...
0
votes
1answer
19 views

Cosine similarity between two folders (1 and 2) with documents, and find the most relevant set of documents (in folder 2) for each doc (in folder 2)

I have one folder named iir, it has 500 txt files. I have another json file named video (with dictionary structure). I wish to compute: for each of the 500 txt files, find the cosine similarity with ...
1
vote
0answers
18 views

Extracting sections from document based on list of keywords - Python

I am new to NLP and I would like to ask how can I extract sentences from the text based on keywords that I have using Python. I created a list of keywords which will be used to extract sentences from ...
0
votes
0answers
5 views

What is a good standard dataset to compare LDA text mining technique to another one I’ve developed?

I developed a topic modeling technique which uses text network analysis to detect topics within a document (or a corpus). I want to compare it to Latent Dirichlet Allocation (LDA). What is a good ...
-1
votes
0answers
23 views

Machine Learning based software vulnerability scoring

A challenge in scoring vulnerabilities detected in web applications and REST-based applications is outright automation of the severity scoring process. While software packages are scored using the ...
0
votes
2answers
51 views

How to select features for Text classification problem

I am working on a problem where we need to classify user query into multiple classes. Problem: Suppose we are running a website for selling products. The ...
0
votes
0answers
16 views

Classifying variable types on a list of variables

I have a list of around 700 variables which I need to perform a variable cleanup on. What complicates things is there are different numeric codes which flag an invalid value and these differ by the ...
0
votes
1answer
19 views

How to implement a basic query management and recommendation system

I'm trying to prototype a system where given a textual query (e.g. a question), I get a list of most relevant documents/questions among a pool of available documents/questions (similar to what we see ...
0
votes
0answers
8 views

Expanding entity extraction to certain types of named-noun phrases (Bi-grams or tri-grams)

I have a corpus of articles from business publications. I'm interesting in extraction information relating to specific types of technologies and processes, as well as just interesting phrases. ...
0
votes
1answer
28 views

Extracting specific portions of text from poorly formatted document?

I have a corpus of text files (free books) which are poorly formatted. The goal is to extract a particular chapter (say chapter 2) from the raw text with all weird formatting removed. Some documents ...
0
votes
1answer
34 views

How to detect phrases from an English sentence.

The question is not about detecting keyphrases. It is about detecting a combination of words makes a valid phrase or not. For example, "John reads New York Times in New York." Here, the phrases ...
0
votes
0answers
17 views

How to handle text encoding while text input or cleaning?

Often while data reading, cleaning or performing similar tasks; often I stumble upon error message such as UnicodeEncodeError, or while tokenising these texts, I get various errors because of encoding....
0
votes
1answer
110 views

Text extraction from documents using NLP or Deep Learning

I am looking for references(Papers/github projects) on how to use deep learning in a text extraction task. Recently I was given a task to extract important information from documents of similar type, ...
0
votes
0answers
24 views

Multiclass Text Classification of Wikipedia Articles using Deep Learning

I have a collection of Wikipedia dumps. I need to classify them in a list of categories that I have. The categories are like, Sports, Law, Music, Movie, etc. There are around 300 categories. I ...
1
vote
0answers
35 views

One Class Classification

I was reading about One Class Classification but had two doubts - 1) How does One Class Classification work because the training data is of only a particular class so in that case, during testing, ...
0
votes
1answer
39 views

NLP - extract sentence parts related to people

Would be grateful for any definitions of the problem/sources about the following problem: given one-two sentences, find out which given adjectives/verbs/partial sentences in the sentence refer to a ...
1
vote
1answer
58 views

Mining of massive datasets

Can someone answer this question: It is from an exercise in the book: Mining of massive datasets: Chapter 3: Finding Similar Itemsets What is the largest number of k-shingles a document of n ...
0
votes
0answers
25 views

Locality Sensitive Hashing for two sets of Documents

I have found implementations of Locality Sensitive Hashing based on Cosine Similarity But They all deal with finding cosine similarity on a single set of documents. But I want to compare one set of ...
0
votes
0answers
29 views

Algorithm to learn and predict typographical errors/other patterns

I am having transaction data, which passes from one channel to other channel. Now in that process, the data may be subjected to some spelling mistakes or typographical errors. For eg suppose data from ...
0
votes
2answers
103 views

Machine learning - Algorithm suggestion for my problem using NLP

I am looking for a machine learning algorithm for my problem. I have a set of sentences like, ...
7
votes
1answer
89 views
1
vote
0answers
33 views

Where can I find a dataset for long sequence text chunking? [closed]

Context: I have documents with reviews of articles that have the following structure: Introduction: a description of the review, dates and metadata that will be discarded. (avg~180 words, std~30 ...
1
vote
0answers
31 views

Data analysis of unstructured data that have non linear Key value position [closed]

I have unstructured data consist of three excel sheet Text contents of 3 tables some how related to each other But there is no linear direct relationship between them e.g First row of table 1 is not ...
2
votes
1answer
22 views

How would knowing spammers email address improve spam detection algorithms?

I am new into Artificial Intelligence field and I am working on the classic example of Spam detection using classification. I am using Naive Bayes algorithm as well as SVM. While working on them it ...
1
vote
0answers
23 views

How to do an automated SWOT analysis with Text Mining [closed]

has somebody an idea/approach how to do text mining for a SWOT analysis? (e.g. sentiment analysis) I need to assign categories (Strengths, Weaknesses, Opportunities, Risks) to words in a document and ...
0
votes
0answers
5 views

How to identify CS related algorithms and research areas effectively?

Given a text I want to accurately identify and classify the algorithms (e.g., support vector machines) and research areas (e.g., computer vision). I am just wondering if we can utilise knowlege-bases ...
3
votes
0answers
30 views

Determine the most important documents for supervised learning

I have somewhat of a general/high level question. Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class ...
1
vote
0answers
22 views

Would it work to label 1/5th of my data and once I learn a classifier to slowly provide the unlabeled data? [closed]

I need to classify MOOC video scripts into one of 5 classes which specify the intent of the sentence, e.g. explanation, example, ...
1
vote
0answers
435 views

How can I extract skills from a resume using python? [closed]

I am looking at how to extract a set of skills from an unstructured resume format using Python. Please note that, though I have a basic idea of NLP, I am completely new to Python.
3
votes
1answer
18 views

Are there trained neural networks, that can distinguish a book's author point from what he stands against?

Let's take some specific book to narrow this example. Atlas shrugged by Ayn Rand. When i'll be saying something like "Ayn Rand's ideas", i'll mean only those, which are clearly stated in the book. ...
2
votes
0answers
22 views

Related words extraction in text processing

I'm working on some aspcet based sentiment analysis project applied on some tweets or reviews, so far I've applied all the known preprocessing methods on my training dataset which is an XML file that ...
-2
votes
1answer
74 views

Best way to extract information from text description and match it with set of words

I have 10k records of data, each record represents a unique product(10k class labels) and its description. For example, "Coffee Maker, this product takes coffee beans and brew it, to make tasty cofe". ...
2
votes
2answers
40 views

Build train data set for natural language text classification?

I have extracted ~550 video scripts (subtitles) from 11 free courses on the Coursera platform. I have pre-processed them in terms of punctuation removal, stop words removal, tokenization, stemming and ...
1
vote
0answers
14 views

Any Hadoop Tool to Annotate Text

I have large text files on HDFS... I would like to label some text in those files to improve text analysis? Do you know of any tool like that?
0
votes
1answer
64 views

What is the proper train data format in LSTM?

I want to train a model to detect wrong word using in sentence. I have 1 million sentences(word base or char base) with different length. Each position(word or char) has a label to indicate it is ...
2
votes
1answer
38 views

Why is spam detection a classification problem and not a class modelling problem

Trying to get my feet wet with machine learning on text. The most common dataset I've seen in this space is the sms dataset with classes ham and spam. And the most common and successful approach ...
0
votes
1answer
125 views

Creating labels for Text classification using keras

I have a text file with information that needs to classified based on keywords. The text file contains many number of paragraphs. And the paragraph contains keywords that we want (lets say salary ...
0
votes
0answers
19 views

Convert Lancaster stems back to one single word based on the stem?

I performed Stemming with Lancaster algorithm from the Python NLTK library, which is the most aggressive one, afaik. Here's an online tool that does this with small texts If I have words ...
0
votes
0answers
27 views

Recommended Modelling Technique for Influencer Marketing Scenario

I have an approximately 90,000 row dataset that has information of social media profiles which has columns for biography, follower count, language spoken, name, username and the label (to identify ...
1
vote
1answer
28 views

Identifying most informative (sub)words/vectors that help classify a sample

I am classifying text using fastText which is a word2vec library that can also create vectors for character level n-grams and I have successfully trained a binary classifier. Now I’d like to see what ...
2
votes
2answers
82 views

Plots with shaded standard deviation

What tools can I use to make a visualization similar to this one? I want to have the mean be bolded and the standard deviation be shaded.