Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Is there any GitHub pull-request dataset with labels information?

I also tried to get the labels for a pull-request using GitHub API, but that did not work as labels are not linked with the pull-request. Any idea? How can I get the PR comments labels for GitHub ...
0
votes
0answers
10 views

showing the quality of test set through a manually labeled representative sample

I have created a dataset of 1 million + data points with the testing set being roughly 180,000. This dataset has been labeled automatically from user-generated tags inherent in the data. Due to the ...
1
vote
1answer
33 views

Creating a valid dataset for obtaining results

I have created a domain-specific dataset, lets say it is relating to python programming topic posts. I have taken data from various places specific to this topic to create positive examples in my ...
0
votes
1answer
22 views

Extract data from facebook

I am learning about social media analysis. I am aware that we can extract the data from twitter using hashtags and API. Ex; If I use #covid19, I will get all tweets ...
0
votes
0answers
18 views

Unable to install R package (“tm”) [closed]

I have R version 4.0.2 and I am getting the following error when I try to install a text mining package ...
2
votes
2answers
64 views

Learn (common) grammar / pattern from set of sample strings?

So I currently have a text pattern detection challenge to solve at work. I am trying to make an outlier detection algorithm for a database, for string columns. For example let's say I have the ...
2
votes
1answer
31 views

is it wrong to use average='weighted' when having only 2 classes?

In the book 'Text Analytics with Python', the author provides model_evaluation_utils.py In the code of the .py he does: ...
0
votes
1answer
12 views

how come accuracy_score recognizes the positive label and precision_score does not?

I am executing this code which works perfectly for me: (I only have 'positive' and 'negative' sentiments): ...
0
votes
0answers
22 views

Algorithm for rectifying incorrect text from OCR

I have an OCR (tesseract) engine running on processing "chyron" text from a live stream news feed. However, the OCR is often inaccurate and ends up with some noisy words: ...
3
votes
1answer
48 views

How to create ROC - AUC curves for multi class text classification problem in Python

I am working on a multiclass text classification problem and trying to plot ROC Curve but no success so far. Tried many solutions available but didn't work. Kindly please someone help me out with the ...
2
votes
2answers
28 views

Build a sentiment model from scratch

I would like to know how I can create a sentiment model from scratch. I have my data, list of texts, with no labels about sentiment. ...
3
votes
1answer
61 views

Decision tree and SVM for text classification - theory

I used 4 classifiers for my text data: NB, kNN, DT and SVM. As for NB and kNN I fully understand how they work with text - how we can count probabilities for all words in NB and how to use similarity ...
2
votes
1answer
26 views

Classify text as logical/ not logical

Can some one advise me direction where to look in.Or some resources. Here is a task: User leaves feed back-text with min 50 characters. I need to check if it's normal human sentences/ word ...
0
votes
0answers
16 views

Finding location to get text from in images

I have the task of pulling text from forms given an image of the form. For example, the form may have the form number at the top left or top center. Knowing this, I am supposed to be able to output ...
1
vote
1answer
45 views

Extracting text into columns

I have a set of customer reports, each is in ms word file. they are all in a similar pattern, for example they start with Name: --, Age: --, Date: --, etc... is there a way to extract particular ...
0
votes
1answer
27 views

How to identify similar words as input from a dictionary

Let's say I have a CSV file (single column) with a list of words as input. Meaning the file looks like below Input Terms ...
1
vote
1answer
27 views

Text comparison: spot the differences

I would like to know what would be the best approach to compare two texts and see the differences between them. For example: ...
0
votes
0answers
7 views

Multi-documents text annotation tool?

I have been looking for a good annotation tool/interface for text annotation. My requirements are as follows: Display of multiple short-text social media posts on the same page, as one task; ...
1
vote
1answer
32 views

Which insights a data scientist could derive from text-analysis? [closed]

I have many texts and I am trying to analyse them. After tokenising them, studying words frequency, spotting any typos, studying punctuations, I have been working on POS tagging. Since it is my first ...
1
vote
2answers
54 views

Extract information using NLP and store it in csv file

I have a text file that stores the pickup, drops, and time. SMS text is a dummy file that is used to train a cab service model. The text is like in this format: ...
2
votes
1answer
31 views

Extract First names from usernames

John10 , michaelscott, James.white , Jr.Jones , James-Anderson ,WhiteWalter10 -- These are some of the different cases of usernames possible(there may be more ). I have about 200K such usernames . I ...
0
votes
0answers
7 views

Dirichlet smoothing as an IDF component

How can Dirichlet smoothing be used as an IDF component to estimate the probabilities of a Topic model ? i.e, Smoothing with a background collection model to estimate topic model ? I've seen many ...
0
votes
1answer
44 views

Why do i get error when I use corpus in Orange v 3.25.0 text mining widget and no import document option?

I am experimenting with Orange data mining tool. When i use 'Corpus' from text mining. It gives me error. . I tried many things, but still unable to resolve this issue. Besides that, In text mining, ...
0
votes
0answers
22 views

How to segregate resume layouts into different types?

I'm looking for any suggestions on how to segregate resume layout into different types. How do one proceed with such a task? I mean resumes are usually available as pdf or docx format and when we ...
0
votes
1answer
25 views

breaking joined words into meaningful ones during text mining

I'm performing an aspect-based sentiment on consumer complaints. I'm tokenizing at the sentence level. ...
1
vote
1answer
12 views

How to represent a document in test data with the Document-Term Matrix created from the training set?

I build a classifier of documents using the vector representation of each document in the training set (i.e a row in the Document-Term Matrix). Now I need to test the model on the test data. But how ...
0
votes
0answers
13 views

How to build a resume text block segmentation algorithm using deep learning?

Assume I have resume, I want to segment different text blocks such as personal info, education , experience etc.., Is is possible to segment text blocks by converting pdf or docx resume into image ...
0
votes
0answers
20 views

English Word popularity scoring

I'm new to data science and looking for the word popularity ranking algorithm. Given a list of words, What would the best and easy to implement algorithm to score each word based on the word ...
0
votes
1answer
21 views

Text Mapping - Medicine Names

We have a problem where we have a standardized database of Medicine names. On the other hand, there is a subset of medicine names which could have spelling mistakes, different structure or hypens, ...
0
votes
0answers
11 views

How to identify corresponding record of a car from a semi-structured string?

I am trying to build an application that can take a record of a car from different websites, compare it to data i have in a CSV file and return me the matching row. Each website will present and ...
0
votes
0answers
12 views

What type of deep learning model should I use to extract fields from legal agreement documents?

I have rent agreement documents and a csv file having output column fields (labels) like agreement amount,name of tenant etc. For every document an AI model should ...
0
votes
0answers
15 views

Benchmark queries for Benchmark dataset with ranked list of documents

I aim to evaluate the ranking of an information-retrieval system. For a benchmark dataset like TREC, I have followed the qrels file which has list of documents for a particular topic (query) with ...
0
votes
1answer
21 views

Get a prediction from the new data inputted against the model, but an error is produced, how to adapt the R code for it to work?

In the R code below, I included the sentences when looking to compare the manually classified with lexicon dictionary results by positive, negative and neutral (in matrixdata1), the algorithms results ...
0
votes
0answers
41 views

How do I extract album and song titles from this plain text file?

Inspired by topic modeling and clustering analysis of Taylor Swift's lyrics, I want to do the same for the band Nightwish. I scraped Dark Lyrics (see script) for all of their lyrics and saved the ...
0
votes
0answers
7 views

Can text analysis approaches using machine learning be used on financial statement reports?

In a paper (Correa et al, 2017), the following is mentioned: " Alternative text analysis methods, such as machine-learning approaches, require some type of classification that would help in ...
0
votes
0answers
13 views

Named Entity Recognition/Linking with a dictionary of names (e.g., move titles, book titles, company names, people names)

I have a lot of text messages on which I want to perform Named Entity Recognition and Named Entity Linking. For example, I want to detect all movie names and find their IMDB ID. To make things easier,...
0
votes
0answers
8 views

Merging (intersecting) more than two posting list in linear time

The intersecting algorithm for two posting lists implemented below: ...
0
votes
1answer
113 views

Does Python have R's tidytext equivalent?

I can't seem to find a tidytext (R library) equivalent in Python. Text mining in Python seems quite weak compared to R.
0
votes
0answers
15 views

Can we use TF-IDF along with Weighted Frequency for Text Summarization?

I've been working on a text summarization problem. After studying this blog, I used weighted-frequencies of words present in a document for summarization. I would compute the weighted frequency ...
0
votes
0answers
17 views

How do I discern document structure from differently-tagged XML documents?

I have a body of PDF documents of differing vintage. Our group had exported the documents as text to feed them into a natural-language parser (I think) to pull out subject-verb-predicate triples. ...
0
votes
1answer
54 views

NLP - Simple approach to identify commonalities in text comments between people

For something we are working on, we were looking for a simple way to compare from review/feedback data against a question (for which there are multiple responses from multiple people), the following: ...
0
votes
0answers
8 views

Looking for suggestions on performing Sementic Analysis of ASR text

Currently I am working on a project where I have ASR on which I am performing semantic analysis to extract meaning out of it. The ASR text contains huge amount of vague conversational text which needs ...
0
votes
1answer
16 views

How to identify new job descriptions/postings from a set of documents when I have a set of already labeled job descriptions/postings

Suppose I have a set of already labeled documents -- some of them are job descriptions/postings (these are documents of interest), and some of them are not. I wonder what kind of method would allow me ...
3
votes
1answer
15 views

Doubt on scope of text classification problem

I have a dataset that describes the sellers who are selling various brands. I need to identify the source (where did he buy those brands he is selling from) of those sellers. (Dimension of dataset 11,...
1
vote
1answer
15 views

how to create a searchable tree on Persian text?

I wanna clean a Persian text from stop-words. I already have stop-word data that is provided on the below link. It seems to me, if I have a pre-built tree on stop-words, I could save lots of time. I ...
0
votes
2answers
29 views

Extract amount from free text

I want to extract various amounts and tenure of contracts from different contract documents that we have. For example : Mr xyz, this contact is valid for 3 Months and you have to pay $3000 as ...
0
votes
0answers
49 views

Get row wise frequency count of words from list in text column pandas

I have a data frame with a Audio Transcript column from customer care phone conversation. I have created one list with words and sentences ...
1
vote
1answer
29 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
1
vote
0answers
24 views

What is the state-of-the-art method/algorithm to extract Keywords from text?

What is the state-of-the-art method/algorithm to extract data from text without regarding the language of the text? The methods I am reading about are Rake, Yake or using LTSM Networks to identify ...
2
votes
1answer
167 views

NER vs Text classification for very short sentences

Given a large set of short sentences (around 20-30 words) and multi label task (around 100 labels , can be to 3 labels per sentences ). The location of each annotation is not impotent (i.e i only ...

1
2 3 4 5
11