Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
1answer
575 views

How to extract entities from text using existing ontologies?

I am working on a entity extraction task and I am using Stanford CoreNLP NER. Here, I want to detect entities of type "Animal", "Building", "Imagery", etc., which are not covered in Stanford CoreNLP ...
0
votes
2answers
2k views

How to alter word2vec wikipedia model for n-grams?

I have a very little data, so my word2vec model does not perform well. My intention is to identify words similar to technical terms such as 'support vector machine', 'machine learning', 'artificial ...
1
vote
0answers
46 views

How to calculate lexical cohension and semantic informaticveness for a given dataset?

In 'Automatic construction of lexicons, taxonomies, ontologies, and other knowledge structures' they have mentioned; There are two slightly different classes of measure: lexical cohesion (sometimes ...
2
votes
2answers
184 views

Are there any measures for Entity Ambiguity?

When using/building a system for Entity Linking, is there a well-known measure for "ambiguity degree" of an entity? Some approach to compare named entities regarding how difficult to disambiguate?
1
vote
2answers
485 views

Text classification- What to do when train and test data have different features

I am performing binary text classification. I have to classify a tweet 0 if neutral and 1 if hate speech. So as general thumb rule i preprocessed my data. create term document frequency and After ...
9
votes
2answers
207 views

what machine/deep learning/ nlp techniques are used to classify a given words as name, mobile number, address, email, state, county, city etc

I am trying to generate an intelligent model which can scan a set of words or strings and classify them as names, mobile numbers, addresses, cities, states, countries and other entities using machine ...
1
vote
1answer
105 views

Data Mining - Intent matching and classification of text

PROBLEM Suppose you have a list of 100,000+ google queries related to travel bookings. For example: ...
1
vote
2answers
107 views

Extract Pattern using Short Text Processing

We are facing issue in our project. We have a data set of around 25000 rows, we have a column name title, it contains text data and we have a score column in the data set.We want to use Machine ...
0
votes
1answer
158 views

Text classification and clustering with complete date imbalance

I have a set of scientific papers of authors who have common research interests from PUBMED and I would like to: Clustering papers and extracting features from them in order to find other authors ...
0
votes
1answer
316 views

Using NLP to detect insurance Fraud [closed]

How can I use Natural Language Processing to detect insurance Fraud? I understand it for the structured part, but I need more insight on what kind of text data will help with detecting fraud, also ...
5
votes
1answer
150 views

How to compute document similarities in case of source codes?

I try to detect the probability of common authorship (person, company) of different kind of source code texts (webpages, program codes). My first idea is to apply the usual NLP tools like any token ...
0
votes
1answer
414 views

RMSE in Weka Time Series Forecasting

I am using Weka Time Series Forecasting to forecast the trend of the topic NLP in 2018. For that I used ...
0
votes
1answer
33 views

How to read table [closed]

I have a .txt file and I am required to read the bottom table using read.table() http://pages.stat.wisc.edu/~jgillett/327-1/4/beef.txt How could I clean the text to do that?
5
votes
2answers
504 views

Clustering Observations by String Sequences (Python/Pandas df)

I have a dataset consisting of approximately 2 million unique observations. It was initially a set of ID's and URLs. The goal is to cluster the ID's based on the URLs looked at. I transformed both ...
2
votes
0answers
160 views

How to cluster sentences based on company names from a post(s) containing several company names using similarity metric.

My corpus contains several posts having text for several companies i.e. each post contains information about several companies. I want to cluster the information based on few company names that I ...
2
votes
1answer
358 views

Text Mining with Naive Bayes

I'm implementing prediction code for courses of computing fields using Naive Bayes classifier. The output is to predict whether the course is (management, design, database, analysis,…9 classes). I ...
1
vote
0answers
81 views

Performance Metric for topic extraction when there is no ground truth

I am extracting topics from text using a predefined ontology containing 2690 concepts, wordnet(to expand concept terms with their synsets, and other morphological forms of the same word) and lucene to ...
1
vote
1answer
17 views

Any research on segmentation of non-text contents out of (mostly) text-documents?

Documents, especially technical ones often contain non-text content in blocks (code snippets, os commands etc). Is there any efficient way to identify the starting and ending lines of those blocks?
2
votes
1answer
2k views

Right Way to Input Text Data in Keras Auto Encoder

I have several thousand text documents and I am currently working on obtaining the latent feature representations of words and generate sentences using variational auto encoder. The main obstacle I am ...
1
vote
1answer
53 views

Finding repeating string patterns in thousands of files

I have files like the following: https://pastebin.com/5mkXY1aU These are created by filling predefined forms so there are thousands of them that would match a pattern. I will try to give more ...
1
vote
0answers
1k views

How to improve the accuracy of Random Forest for Text Categorization

I am working on a text Categorization problem, the objective is to classify related companies into their corresponding categories. This is a single category classification problem and not multi-class ...
3
votes
1answer
3k views

What are useful evaluation metrics used in machine learning

I am using CNN in order to predict codes after analyzing text. As an example, I will write "I am crazy" .. the model will predict some code " X321". All this based on CNN. I want to evaluate my ...
2
votes
1answer
274 views

Getting unexpected result while using CountVectorizer()

I am trying to use CountVectorizer() in a loop, But I am getting an unexpected result. On the other hand, if I use it outside the loop then it works fine. I believe ...
3
votes
2answers
2k views

How to use different classes of words in CountVectorizer()

Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example Text #1 : Comedy 10%, Horror 50%, Romance 1% Text #2 : ...
0
votes
4answers
497 views

Ways to convert textual data to numerical data

I've been looking for ways to wrangle my data which contains both text and numerical attributes. There are of course several algorithms for numerical data, but I am looking for suggestions regarding ...
1
vote
1answer
56 views

Classify phrases as biomedical or non-biomedical

Words like myotonic are biomedical, but words like new appear in regular English texts. I can imagine a few ways to classify ...
2
votes
1answer
72 views

How to estimate probabilities of different classes for a Text

Suppose I have a piece of writing and I want to assign probabilities to different genres (classes) based on its contents. For example Text #1 : Comedy 10%, Drama 50%, Fiction 20%, Romance 1%, ...
3
votes
0answers
149 views

How does Api.ai Google dialogueflow classifies “intent” and extracts data from slots

I am trying to build a very naive version of Api.ai, now Google DailogueFlow. I wanted to know two things. How DF classifies sentences with entities in it that can be user created and/or things like ...
2
votes
3answers
5k views

Text mining for text matching

I m new in text analysis and need your advice to help medical students to write properly and correctly. The students describe sicknesses as they observe them; however, they must use an "official ...
1
vote
2answers
96 views

Sentiment analysis using sources other than the IMDB data

I'm curious as to whether the training data for a sentiment analysis tool needs to be specifically geared toward the domain it's being using in. For example, the IMDB movie review data makes sense if ...
1
vote
1answer
118 views

How to extract specific parts of text from a string? [closed]

For example: Here is a textual input: "ALL imagery SINCE 1952 20 MULE T aerial BOOSTER & Multi-Purpose Neutralizer.MAY BE “AHMFLL |-E SWALL """""" -5 NETWT4LBS1 DZ (65 OZ) 1.84 kg " Output ...
1
vote
1answer
271 views

Converting similarity matrix before inputting to t-sne

I have a cosine similarity matrix where I want to adjust it to inputto t-sne. I fond the following explanation in a FAQ. As mentioned there I have made the diagonals to zero. what does it mean by <...
2
votes
1answer
96 views

Datasets in NLP research papers

I am looking for a dataset containing a large number of NLP research papers and abstracts. Are there any open access datasets like that? If so, can you please share the details?
0
votes
2answers
212 views

Unsupervised text clustering using a driving list

I want to apply unsupervised clustering on a set of short texts, which I need to divide into 2 clusters. Also I know that one of my clusters is likely to contain some words (non-exhaustive list) and ...
2
votes
1answer
194 views

Facing a difficult regular expression issue in cleaning text data

I am trying to substitute a sequence of words with some symbols from a long string appearing in multiple documents. As an example, suppose I want to remove: ...
1
vote
2answers
485 views

Cluster titles or ingredients of food into n-categories

I have a dataset which has information about food recipes (in german), that looks like this: Here is a link to a small .csv file (first 1000 rows of my data) https://drive.google.com/file/d/...
2
votes
1answer
3k views

How does keras calculate accuracy for multi label classification?

I am using this code for a multilabel problem classification. ...
4
votes
1answer
469 views

Word embedding vectors for keyphrase extraction

I am just interested to know if we can use the generated word embedding vectors to extract keywords from a document or not. If yes, how?
5
votes
2answers
434 views

Topic modeling for short length sentences

I have a graph which was already separated into clusters. Each node in the graph has a label (typically, it's a function's name like ...
1
vote
1answer
48 views

After choosing top models in classification? Can I apply it on the rest of my dataset

I am working with a corpus that has 5 datasets in product reviews (A, B, C, D and E), mine is a text classification problem and I need to find the best 5 top models in terms of classification ...
0
votes
1answer
53 views

Filter unwanted terms

I have the following keywords retrieved from a text document. ...
0
votes
2answers
340 views

Rank terms in a bag -of-words model

I have a set of documents where I need to extract important keywords in the document and then rank those keywords. The ranking should be done based on relevance and/or other metrics. Are there any ...
0
votes
1answer
2k views

Better input for Doc2Vec

I want to perform Doc2Vec on a twitter dataset. As each tweet consists of a nummber of special characters ,numbers, urls, mentions and hashtags, non-english words, what should be my input for Doc2Vec? ...
0
votes
1answer
355 views

Text Mining from Images

I want to do the following project and I think the best way is using tensorflow/keras: I have photos as images from a list with prices of products and I need classify these text objects from the ...
0
votes
1answer
164 views

Can someone explain the lambda collocation metric?

I'm trying to understand the lambda collocation scoring metric from the quanteda package in R, but the explanation in the documentation is incredibly difficult to understand. Can anyone explain it ...
1
vote
2answers
256 views

Text post-processing

I have set of newspaper articles and I use TextRank algorithms to identify their keywords to perform a classification. Apart from the important informative keywords, I am also receiving garbage ...
1
vote
4answers
2k views

Parameter Tuning by Cross Validation for Random Forest

I train a binary random forest classifier on scikit-learn's 20 newsgroups dataset. I want to tune the parameters and try so by gridsearch and 3-fold cross validation on the training data. Is there ...
0
votes
1answer
52 views

Identify important less frequent words

I have set of newspaper articles. I want to identify important less frequent words in the set of newspaper articles. Currently I am using TF-IDF scores. However, it does not seem to be a good metric ...
2
votes
3answers
2k views

Text Classifier with multiple bag-of-words

I am training an email classifier from a dataset with separate columns for both the subject line and the content of the email itself. I've pre-processed the content column in such a way that the ...
2
votes
1answer
548 views

Handling data imbalance and class number for classification

1 Is there a way to handle data imbalance? ie if data in each class for training is not balanced, say some classes have 50 documents some other have 200 documents. How to handle this? 2 How to handle ...