We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
1
vote
0answers
2k views

How can I extract skills from a resume using python? [closed]

I am looking at how to extract a set of skills from an unstructured resume format using Python. Please note that, though I have a basic idea of NLP, I am completely new to Python.
3
votes
1answer
20 views

Are there trained neural networks, that can distinguish a book's author point from what he stands against?

Let's take some specific book to narrow this example. Atlas shrugged by Ayn Rand. When i'll be saying something like "Ayn Rand's ideas", i'll mean only those, which are clearly stated in the book. ...
2
votes
0answers
29 views

Related words extraction in text processing

I'm working on some aspcet based sentiment analysis project applied on some tweets or reviews, so far I've applied all the known preprocessing methods on my training dataset which is an XML file that ...
-2
votes
1answer
822 views

Best way to extract information from text description and match it with set of words

I have 10k records of data, each record represents a unique product(10k class labels) and its description. For example, "Coffee Maker, this product takes coffee beans and brew it, to make tasty cofe". ...
2
votes
2answers
109 views

Build train data set for natural language text classification?

I have extracted ~550 video scripts (subtitles) from 11 free courses on the Coursera platform. I have pre-processed them in terms of punctuation removal, stop words removal, tokenization, stemming and ...
1
vote
0answers
17 views

Any Hadoop Tool to Annotate Text

I have large text files on HDFS... I would like to label some text in those files to improve text analysis? Do you know of any tool like that?
0
votes
1answer
268 views

What is the proper train data format in LSTM?

I want to train a model to detect wrong word using in sentence. I have 1 million sentences(word base or char base) with different length. Each position(word or char) has a label to indicate it is ...
2
votes
1answer
177 views

Why is spam detection a classification problem and not a class modelling problem

Trying to get my feet wet with machine learning on text. The most common dataset I've seen in this space is the sms dataset with classes ham and spam. And the most common and successful approach ...
0
votes
1answer
1k views

Creating labels for Text classification using keras

I have a text file with information that needs to classified based on keywords. The text file contains many number of paragraphs. And the paragraph contains keywords that we want (lets say salary ...
1
vote
1answer
61 views

Identifying most informative (sub)words/vectors that help classify a sample

I am classifying text using fastText which is a word2vec library that can also create vectors for character level n-grams and I have successfully trained a binary classifier. Now I’d like to see what ...
4
votes
2answers
2k views

Plots with shaded standard deviation

What tools can I use to make a visualization similar to this one? I want to have the mean be bolded and the standard deviation be shaded.
1
vote
0answers
165 views

Extracting date, relation and noun phrase from text

A sentence (Segmented from a document) as below: ...
7
votes
3answers
142 views

How to plot clusters in nice a way?

I have a large text dataset clusterized. Each cluster is represented by a centroid of the vectorized texts that belong to it, the number of texts, the created date, and other parameters. I can't plot ...
1
vote
0answers
1k views

Keyword Extraction from a text followed by a key value using tensorflow

I have a pdf file that contains information . I would like to extract few key terms/phrase along with a value for example (current balance : CHF (swiss francs) 1,000...
1
vote
2answers
29 views

Determining helpful Amazon feedback

I am in an ML course and one of our tasks is to predict the helpfulness of Amazon reviews. Currently, I am doing what most people seem to . That is, a hashvectorizer(2-gram) on the text, tfidf, ...
0
votes
1answer
79 views

Measuring document cluster cohesion

I have a set of clusters which each cluster contains a list of short documents. I want to compute how coherent and cohesive each cluster is and filter out the incoherent and in-cohesive ones. I am ...
1
vote
1answer
599 views

Euclidean vs. cosine similarity

I have a text dataset which I vectorize using a tfidf technique and now in order to make a cluster analysis I am measuring distances between these vector representations. I have found that a common ...
1
vote
0answers
32 views

Find words related to high or low score

I am working on text analysis problem. Person X can log in his goals and his actions to achieve his goal. Also their score is calculated based on some formula to measure progress of the goal. For ...
1
vote
0answers
41 views

Build a relevancy scoring model of articles using NLP

I'm really new to Data Science and text mining. I want to build a relevancy scoring model. Suppose I have a bag of words (guns, military, terrorists). I also have a list of articles. I want to find if ...
-3
votes
1answer
61 views

german gunning fog index function

I would like to analyse some text and most of my Reviews are german. Does anyone know if python has a good gunning fog index function for german language? I couldnt find anything best regards
2
votes
1answer
54 views

Extract details from bibliometrics data

I have set of bibliometrics data (references). I want to extract the author names, title and the name of the conference/journal from it. Since the referencing style used by different papers vary, I am ...
5
votes
2answers
114 views

Online news classification

I am performing an online news classification. The idea is to recognize group of news of the same topic. My algorithm has these steps: 1) I go through a group of feeds from news sites and I recognize ...
2
votes
2answers
946 views

How to feed my JSON dataset in Keras for character level text classification

I have a JSON dataset, for example: ...
0
votes
1answer
610 views

How to extract entities from text using existing ontologies?

I am working on a entity extraction task and I am using Stanford CoreNLP NER. Here, I want to detect entities of type "Animal", "Building", "Imagery", etc., which are not covered in Stanford CoreNLP ...
0
votes
2answers
2k views

How to alter word2vec wikipedia model for n-grams?

I have a very little data, so my word2vec model does not perform well. My intention is to identify words similar to technical terms such as 'support vector machine', 'machine learning', 'artificial ...
1
vote
0answers
47 views

How to calculate lexical cohension and semantic informaticveness for a given dataset?

In 'Automatic construction of lexicons, taxonomies, ontologies, and other knowledge structures' they have mentioned; There are two slightly different classes of measure: lexical cohesion (sometimes ...
2
votes
2answers
200 views

Are there any measures for Entity Ambiguity?

When using/building a system for Entity Linking, is there a well-known measure for "ambiguity degree" of an entity? Some approach to compare named entities regarding how difficult to disambiguate?
1
vote
2answers
549 views

Text classification- What to do when train and test data have different features

I am performing binary text classification. I have to classify a tweet 0 if neutral and 1 if hate speech. So as general thumb rule i preprocessed my data. create term document frequency and After ...
9
votes
2answers
227 views

what machine/deep learning/ nlp techniques are used to classify a given words as name, mobile number, address, email, state, county, city etc

I am trying to generate an intelligent model which can scan a set of words or strings and classify them as names, mobile numbers, addresses, cities, states, countries and other entities using machine ...
1
vote
1answer
111 views

Data Mining - Intent matching and classification of text

PROBLEM Suppose you have a list of 100,000+ google queries related to travel bookings. For example: ...
1
vote
2answers
108 views

Extract Pattern using Short Text Processing

We are facing issue in our project. We have a data set of around 25000 rows, we have a column name title, it contains text data and we have a score column in the data set.We want to use Machine ...
0
votes
1answer
164 views

Text classification and clustering with complete date imbalance

I have a set of scientific papers of authors who have common research interests from PUBMED and I would like to: Clustering papers and extracting features from them in order to find other authors ...
0
votes
1answer
349 views

Using NLP to detect insurance Fraud [closed]

How can I use Natural Language Processing to detect insurance Fraud? I understand it for the structured part, but I need more insight on what kind of text data will help with detecting fraud, also ...
5
votes
1answer
154 views

How to compute document similarities in case of source codes?

I try to detect the probability of common authorship (person, company) of different kind of source code texts (webpages, program codes). My first idea is to apply the usual NLP tools like any token ...
0
votes
1answer
518 views

RMSE in Weka Time Series Forecasting

I am using Weka Time Series Forecasting to forecast the trend of the topic NLP in 2018. For that I used ...
0
votes
1answer
35 views

How to read table [closed]

I have a .txt file and I am required to read the bottom table using read.table() http://pages.stat.wisc.edu/~jgillett/327-1/4/beef.txt How could I clean the text to do that?
5
votes
2answers
599 views

Clustering Observations by String Sequences (Python/Pandas df)

I have a dataset consisting of approximately 2 million unique observations. It was initially a set of ID's and URLs. The goal is to cluster the ID's based on the URLs looked at. I transformed both ...
2
votes
0answers
169 views

How to cluster sentences based on company names from a post(s) containing several company names using similarity metric.

My corpus contains several posts having text for several companies i.e. each post contains information about several companies. I want to cluster the information based on few company names that I ...
2
votes
1answer
376 views

Text Mining with Naive Bayes

I'm implementing prediction code for courses of computing fields using Naive Bayes classifier. The output is to predict whether the course is (management, design, database, analysis,…9 classes). I ...
1
vote
0answers
85 views

Performance Metric for topic extraction when there is no ground truth

I am extracting topics from text using a predefined ontology containing 2690 concepts, wordnet(to expand concept terms with their synsets, and other morphological forms of the same word) and lucene to ...
1
vote
1answer
17 views

Any research on segmentation of non-text contents out of (mostly) text-documents?

Documents, especially technical ones often contain non-text content in blocks (code snippets, os commands etc). Is there any efficient way to identify the starting and ending lines of those blocks?
2
votes
1answer
2k views

Right Way to Input Text Data in Keras Auto Encoder

I have several thousand text documents and I am currently working on obtaining the latent feature representations of words and generate sentences using variational auto encoder. The main obstacle I am ...
1
vote
1answer
62 views

Finding repeating string patterns in thousands of files

I have files like the following: https://pastebin.com/5mkXY1aU These are created by filling predefined forms so there are thousands of them that would match a pattern. I will try to give more ...
1
vote
0answers
1k views

How to improve the accuracy of Random Forest for Text Categorization

I am working on a text Categorization problem, the objective is to classify related companies into their corresponding categories. This is a single category classification problem and not multi-class ...
3
votes
1answer
4k views

What are useful evaluation metrics used in machine learning

I am using CNN in order to predict codes after analyzing text. As an example, I will write "I am crazy" .. the model will predict some code " X321". All this based on CNN. I want to evaluate my ...
2
votes
1answer
320 views

Getting unexpected result while using CountVectorizer()

I am trying to use CountVectorizer() in a loop, But I am getting an unexpected result. On the other hand, if I use it outside the loop then it works fine. I believe ...
3
votes
2answers
2k views

How to use different classes of words in CountVectorizer()

Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example Text #1 : Comedy 10%, Horror 50%, Romance 1% Text #2 : ...
0
votes
4answers
529 views

Ways to convert textual data to numerical data

I've been looking for ways to wrangle my data which contains both text and numerical attributes. There are of course several algorithms for numerical data, but I am looking for suggestions regarding ...
1
vote
1answer
56 views

Classify phrases as biomedical or non-biomedical

Words like myotonic are biomedical, but words like new appear in regular English texts. I can imagine a few ways to classify ...
2
votes
1answer
76 views

How to estimate probabilities of different classes for a Text

Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example Text #1 : Comedy 10%, Drama 50%, Fiction 20%, Romance 1%, ...