Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
1
vote
1answer
47 views

Build text complexity model based on complex examples

I try to build the user specific model which predicts whether arbitrary English text is complex for particular user or not. Having the complex and easy text samples allows to build such model but what ...
2
votes
2answers
619 views

Extracting sections from document based on list of keywords - Python

I am new to NLP and I would like to ask how can I extract sentences from the text based on keywords that I have using Python. I created a list of keywords which will be used to extract sentences from ...
2
votes
1answer
46 views

Should bag of words in training set include test set data when doing text classification?

I'm doing text classification to identify 'attacks' from Wikipedia comments using a simple bag of words model and a linear SVM classifier. Because of class imbalance, I'm using the F1 score as my ...
8
votes
4answers
3k views

How to deal with spelling errors NLP

I have some data where the main column is the description of one product. The main task is to extract the name of some product from this column, where it sometimes is spelled wrong and amended in ...
1
vote
2answers
22 views

Automatically finding business opportunities in text documents

I am new to machine learning and NLP. I am exploring the possibility of using one of these approaches to automatically examine a large collection of text documents and determine, first of all, if they ...
0
votes
1answer
26 views

Text clustering model on small dataset

Is there any way to run any clustering model on a small dataset with 290 text records (minimum character size is 100)?
1
vote
1answer
54 views

Emotional tension score in sentences

I am beginner in natural language processing and my goal is to find a way to score sentences based on their emotional tension. More specifically, I would like to know to what degree a sentence ...
0
votes
0answers
12 views

Keyword Imputation Using Machine Learning and NLP

I am fairly new to machine learning and data imputation so forgive me if I am using wrong words or not feasible ideas. I have a table as follows, with text in the Headline column and a list of ...
1
vote
0answers
7 views

I'm looking for some scripts or tools that can take some text, understand and generate multichoice questions, cloze deletion, pop quiz [closed]

I'm a fairly entry level coder in python but I was hoping for some guidance on if, or how I could load a block of text to generate random questions to test comprehension. My goal is to conduct a large ...
1
vote
0answers
44 views

How to extract numerical information from text descriptions

I have an attribute that is the description of an operation (i.e description of a building consent), I need to translate this to a mathematical operation. I need to find out the new number of dwelling ...
2
votes
1answer
384 views

Get row wise frequency count of words from list in text column pandas

I have a data frame with a Audio Transcript column from customer care phone conversation. I have created one list with words and sentences ...
0
votes
1answer
47 views

Text mining in Amazon product review using R. I wasn't able to extract the particular product's review

Text mining on Amazon product review using R Program. I wasn't able to extract the particular product's review(i.e.If iphone 11 has 6k review, I need to extract all of it.) I'm getting only one column ...
0
votes
0answers
21 views

NLP conversation data - Pre-processing steps

I have text data, so data that has been transcribed from conversations from employees to customers. So each call has a recording that has been transcribed to text. I am looking to do some analytics ...
3
votes
1answer
522 views

Product classification in hierarchical categories based on multiple parameters and non-standard descriptions

I want to start a machine learning project in my company and a really big pain for spend analysts is to classify the products that buyers order for maintenance, tooling, raw material and such, as the ...
1
vote
1answer
59 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
1
vote
1answer
107 views

Information Extraction/Semantic Search for long, unstructured documents

I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store ...
2
votes
0answers
22 views

Unstructured data not template based to structured data

I am working on a project where my goal is to extract data from a large diversity of pdf that does not follow a template (i.e unstructured data not template based). My ORC part works well and now I am ...
0
votes
1answer
90 views

What is the best approach to perform information extraction from tourist reviews using NLP, DL?

I am interested in performing some information extraction from tourist reviews about different places. I have data of 50 different places and around 300-400 reviews about each of them and I would like ...
1
vote
1answer
21 views

Methods for finding characteristic words for a group of documents in comparison to another group of documents?

I'm working on a problem of anomaly detection, where at the end of the anomaly detection I will have a group of documents consisting of a title of each object that was flagged as anomalous. At the ...
0
votes
0answers
20 views

Problem Direction - Text Data - Conversation Classification

The problem I have a problem where I have text data that has been transcribed from a conversation. These conversations have been marked as a pass or fail in terms of compliance by a person, ie they ...
2
votes
2answers
39 views

How to stop a text-classification model from depending on only couple of the words from input text instead of entire sentence?

I have a text classification deep-learning model, which takes in a text and outputs a softmax probability. I am using glove embeddings to represent my input text in numerical form for the DL model. ...
3
votes
1answer
780 views

Is there an algorithm or NN to match two documents, basically not closely similar?

Is there an algorithm or NN to match two documents? One is a claim description (e.g. a CV or product offer) and another is a requirements description (e.g. vacancy description or RFP). They are not ...
1
vote
3answers
1k views

NLP to detect duplicates for very technical language

I have the following scenario, to detect duplicate products based on the description fields. The Description Field contains product technical name, dimensions, characteristics. My model needs to ...
2
votes
1answer
45 views

Classify documents using a set of known vocabularies

I have a bunch of documents that I want to classify which ones talk about soccer (unsupervised learning, I do not want to manually label the documents). One way I am thinking about is to go online ...
4
votes
1answer
71 views

Determine the most important documents for supervised learning

I have somewhat of a general/high level question. Assume I'm doing supervised machine learning on some text data (tweets for example) and categorizing the documents to a certain taxonomy (multi-class ...
10
votes
3answers
19k views

Public dataset for news articles with their associated categories

I am wondering if there are any public datasets of Google news with various news categories such as politics, entertainment, lifestyle, general news, sports etc. I want to use such dataset for topic ...
1
vote
1answer
406 views

Hellinger Distance in Gensim

I have set of documents as follows where each document has set of words that represents the content of it. ...
2
votes
1answer
35 views

How do I split contents in a text that would include two or more different themes (context) in NLP?

For example, a text: "The airlines have affected by Corona since march 2020 a crime has been detected in Noia village this morning" the output should be: The airline companies have affected ...
1
vote
1answer
39 views

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them?

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them? That is, to get results which say that, for example, the ...
1
vote
1answer
552 views

Sentiment Analysis of News Headlines

I'm trying to do sentiment analysis of News Headlines about a particular subject mentioned in it. Initially, I used TextBlob library for sentiment analysis to ...
1
vote
1answer
20 views

how to create a searchable tree on Persian text?

I wanna clean my huge text data from stop-words. I already have stop-word data that is provided on the below link. It seems to me, if I have a pre-built tree on stop-words, I could save lots of time. ...
0
votes
1answer
33 views

Text Mapping - Medicine Names

We have a problem where we have a standardized database of Medicine names. On the other hand, there is a subset of medicine names which could have spelling mistakes, different structure or hypens, ...
0
votes
1answer
28 views

How to identify the feature that make the model misclassifed in text classification

Hi I am working on social media financial THAI text classification, the problem with this one is the confused classes, the misclassified prediction has a pattern that consistent as a pair. and I want ...
0
votes
2answers
53 views

Determining whether a sentence is "cliche" using NLP

I have a collection of essays from students. Each essay is about the same topic and of the same word length. My goal is to develop a machine learning algorithm that pinpoints "cliche" ...
1
vote
1answer
37 views

Document ranking on a web scraped dataset without any labelled data

I want to create a document ranking model which returns similar rows in the dataset for a sample query. The text in this corpus is standard english but without any labels (ie no query-related ...
1
vote
1answer
53 views

How to calculate lexical cohension and semantic informaticveness for a given dataset?

In 'Automatic construction of lexicons, taxonomies, ontologies, and other knowledge structures' they have mentioned; There are two slightly different classes of measure: lexical cohesion (sometimes ...
2
votes
1answer
38 views

Clustering Strings Based on Similar Word Sequences

In my dataset I have a feature having below data : Input Feature Brain Dementia Routine(Comfortone) Morning Check Dementia Brain-Routine(Comfortone) Brain MRA Routine (Comfortone) Brain-Dementia/...
0
votes
2answers
488 views

Apply Labeled LDA on large data

I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
0
votes
1answer
47 views

Inference from text data without label or Target

I have a use case where I have text data entered by an approver while approving of some loan. I have to make some inferences as to what could be the reasons for approval using NLP. How should I go ...
3
votes
1answer
646 views

How to choose threshold for gensim Phrases when generating bigrams?

I'm generating bigrams with from gensim.models.phrases, which I'll use downstream with TF-IDF and/or gensim.LDA ...
0
votes
2answers
66 views
3
votes
1answer
49 views

How to segment old digitized newspapers into articles

I'm working on a large corpus of french daily newspapers from the 19th century that have been digitized and where the data are in the form of raw OCR text files (one text file per day). In terms of ...
0
votes
1answer
25 views

how to filter out and discard irrelevant tweets in simplest way possible

I have lot of tweets and from which i need to filter out and discard irrelevant tweets. the criteria for a tweet to be irrelevant is very simple. if all that a tweet has is emojis or a single hastag ...
1
vote
1answer
114 views

NLP - Simple approach to identify commonalities in text comments between people

For something we are working on, we were looking for a simple way to compare from review/feedback data against a question (for which there are multiple responses from multiple people), the following: ...
0
votes
2answers
33 views

How to stay up to date in NLP and use the best approaches?

There are many fast advancements in NLP field, BERT, RoBERTa, ALBERT, and XLNe, and no one can check the news or papers daily. Is there any way or site that keeps track of all these new developments ...
1
vote
1answer
84 views

Recurrent Neural Networks Over Multiple Documents Over Time

So in my head, I have an idea about what this architecture should look like, or at least behave, but I am having trouble implementing it. So let me describe the problem, and if anyone has an idea on ...
3
votes
1answer
71 views

Extract details from bibliometrics data

I have set of bibliometrics data (references). I want to extract the author names, title and the name of the conference/journal from it. Since the referencing style used by different papers vary, I am ...
1
vote
1answer
51 views

Suggestion for a better way to organize data to generate frequent item-sets?

I have a data of a bag of words in a document. The data has 3 columns: {document number, word number, count of the word in the number}. I am supposed to generate ...
0
votes
0answers
7 views

How to work out the optimum threshold for BIRCH clustering

I am working with a large dataset so I thought that BIRCH would be an ideal clustering algorithm to apply. Can anyone suggest how I can work out what the optimum threshold would be?
1
vote
0answers
14 views

Choice of the number of topics (clusters) in textual data

I have a social science background and I'm doing a text mining project. I'm looking for advice about the choice of the number of topics/clusters when analyzing textual data. In particular, I'm ...

1
2 3 4 5
12