Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
1
vote
1answer
11 views

How to represent a document in test data with the Document-Term Matrix created from the training set?

I build a classifier of documents using the vector representation of each document in the training set (i.e a row in the Document-Term Matrix). Now I need to test the model on the test data. But how ...
1
vote
1answer
17 views

Text comparison: spot the differences

I would like to know what would be the best approach to compare two texts and see the differences between them. For example: ...
0
votes
0answers
4 views

Multi-documents text annotation tool?

I have been looking for a good annotation tool/interface for text annotation. My requirements are as follows: Display of multiple short-text social media posts on the same page, as one task; ...
0
votes
1answer
21 views

Text Mapping - Medicine Names

We have a problem where we have a standardized database of Medicine names. On the other hand, there is a subset of medicine names which could have spelling mistakes, different structure or hypens, ...
1
vote
2answers
45 views

Extract information using NLP and store it in csv file

I have a text file that stores the pickup, drops, and time. SMS text is a dummy file that is used to train a cab service model. The text is like in this format: ...
1
vote
1answer
30 views

Which insights a data scientist could derive from text-analysis? [closed]

I have many texts and I am trying to analyse them. After tokenising them, studying words frequency, spotting any typos, studying punctuations, I have been working on POS tagging. Since it is my first ...
1
vote
1answer
27 views

Extract First names from usernames

John10 , michaelscott, James.white , Jr.Jones , James-Anderson ,WhiteWalter10 -- These are some of the different cases of usernames possible(there may be more ). I have about 200K such usernames . I ...
0
votes
0answers
4 views

Dirichlet smoothing as an IDF component

How can Dirichlet smoothing be used as an IDF component to estimate the probabilities of a Topic model ? i.e, Smoothing with a background collection model to estimate topic model ? I've seen many ...
0
votes
2answers
763 views

How to extract entities from text using existing ontologies?

I am working on a entity extraction task and I am using Stanford CoreNLP NER. Here, I want to detect entities of type "Animal", "Building", "Imagery", etc., which are not covered in Stanford CoreNLP ...
0
votes
1answer
20 views

Why do i get error when I use corpus in Orange v 3.25.0 text mining widget and no import document option?

I am experimenting with Orange data mining tool. When i use 'Corpus' from text mining. It gives me error. . I tried many things, but still unable to resolve this issue. Besides that, In text mining, ...
1
vote
1answer
29 views

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them?

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them? That is, to get results which say that, for example, the ...
1
vote
2answers
936 views

extraction information from resume

I have a project in machine learning in which I need to analyze a curriculum vitae. for that I have to write a python program. It uses basic techniques of Natural Language Processing like word ...
2
votes
1answer
37 views

How to segment old digitized newspapers into articles

I'm working on a large corpus of french daily newspapers from the 19th century that have been digitized and where the data are in the form of raw OCR text files (one text file per day). In terms of ...
0
votes
1answer
45 views

Does Python have R's tidytext equivalent?

I can't seem to find a tidytext (R library) equivalent in Python. Text mining in Python seems quite weak compared to R.
0
votes
2answers
256 views

Apply Labeled LDA on large data

I'm using a dataset contains about 1.5M document. Each document comes with some keywords describing the topics of this document(Thus multi-labelled). Each document belongs to some authors(not just one ...
0
votes
1answer
52 views

NLP - Simple approach to identify commonalities in text comments between people

For something we are working on, we were looking for a simple way to compare from review/feedback data against a question (for which there are multiple responses from multiple people), the following: ...
0
votes
2answers
29 views

NLP: Getting the top 5 or top 10 predictions

I am working on a social networking application and I have to make its news feed better. For example: If someone searches for 'suggest me some good books', it should yield some names. Now, I have ...
0
votes
0answers
22 views

How to segregate resume layouts into different types?

I'm looking for any suggestions on how to segregate resume layout into different types. How do one proceed with such a task? I mean resumes are usually available as pdf or docx format and when we ...
0
votes
1answer
24 views

breaking joined words into meaningful ones during text mining

I'm performing an aspect-based sentiment on consumer complaints. I'm tokenizing at the sentence level. ...
0
votes
0answers
12 views

How to build a resume text block segmentation algorithm using deep learning?

Assume I have resume, I want to segment different text blocks such as personal info, education , experience etc.., Is is possible to segment text blocks by converting pdf or docx resume into image ...
0
votes
0answers
17 views

English Word popularity scoring

I'm new to data science and looking for the word popularity ranking algorithm. Given a list of words, What would the best and easy to implement algorithm to score each word based on the word ...
5
votes
1answer
364 views

Find matching text from a text column

This is my first time to use Data Analytics tool to figure out a solution to a problem. I have a table with following columns ...
0
votes
1answer
27 views

Association Rule Mining across two market baskets

I am quite familiar with Association Rule mining but I need to use it to associate ACROSS two market baskets instead of finding support WITHIN a market basket. Imagine customers come to a Store A ...
1
vote
1answer
49 views

Recurrent Neural Networks Over Multiple Documents Over Time

So in my head, I have an idea about what this architecture should look like, or at least behave, but I am having trouble implementing it. So let me describe the problem, and if anyone has an idea on ...
2
votes
1answer
61 views

Extract details from bibliometrics data

I have set of bibliometrics data (references). I want to extract the author names, title and the name of the conference/journal from it. Since the referencing style used by different papers vary, I am ...
0
votes
0answers
9 views

How to identify corresponding record of a car from a semi-structured string?

I am trying to build an application that can take a record of a car from different websites, compare it to data i have in a CSV file and return me the matching row. Each website will present and ...
0
votes
0answers
8 views

What type of deep learning model should I use to extract fields from legal agreement documents?

I have rent agreement documents and a csv file having output column fields (labels) like agreement amount,name of tenant etc. For every document an AI model should ...
0
votes
0answers
14 views

Benchmark queries for Benchmark dataset with ranked list of documents

I aim to evaluate the ranking of an information-retrieval system. For a benchmark dataset like TREC, I have followed the qrels file which has list of documents for a particular topic (query) with ...
2
votes
1answer
28 views

Attitude to text mining and preparing tokens, irrelevant words, low accuracy

For purpose of quite big project I am doing a text mining on some documents. My steps are quite common: All to lower case Tokenization Stop list and stop words Lemmatizaton Stemming Some other ...
3
votes
3answers
5k views

How to measure the similarity between two text documents?

Assume, I have 100 text documents, and I want to cluster those documents. The first step is the construct pairwise similarity matrix 100X100 for the documents My ...
0
votes
1answer
20 views

Get a prediction from the new data inputted against the model, but an error is produced, how to adapt the R code for it to work?

In the R code below, I included the sentences when looking to compare the manually classified with lexicon dictionary results by positive, negative and neutral (in matrixdata1), the algorithms results ...
1
vote
1answer
80 views

Data Entry Automation with ML

I am working on a data entry task with approximately 6000 entries to go over. The source comes in the form of a string and can look something like this: Air Canada B737 FFS From this I can ...
0
votes
0answers
35 views

How do I extract album and song titles from this plain text file?

Inspired by topic modeling and clustering analysis of Taylor Swift's lyrics, I want to do the same for the band Nightwish. I scraped Dark Lyrics (see script) for all of their lyrics and saved the ...
1
vote
1answer
423 views

Extracting sections from document based on list of keywords - Python

I am new to NLP and I would like to ask how can I extract sentences from the text based on keywords that I have using Python. I created a list of keywords which will be used to extract sentences from ...
0
votes
0answers
5 views

Can text analysis approaches using machine learning be used on financial statement reports?

In a paper (Correa et al, 2017), the following is mentioned: " Alternative text analysis methods, such as machine-learning approaches, require some type of classification that would help in ...
5
votes
2answers
3k views

Clustering or classifing n-gram-based text categories

I have large set of data records looking like this: "text", "category" I extract n-grams from text (2-, 3- and 4-grams) and store count of each n-gram per ...
6
votes
1answer
172 views

How to compute document similarities in case of source codes?

I try to detect the probability of common authorship (person, company) of different kind of source code texts (webpages, program codes). My first idea is to apply the usual NLP tools like any token ...
0
votes
0answers
11 views

Named Entity Recognition/Linking with a dictionary of names (e.g., move titles, book titles, company names, people names)

I have a lot of text messages on which I want to perform Named Entity Recognition and Named Entity Linking. For example, I want to detect all movie names and find their IMDB ID. To make things easier,...
0
votes
0answers
7 views

Merging (intersecting) more than two posting list in linear time

The intersecting algorithm for two posting lists implemented below: ...
0
votes
0answers
12 views

Can we use TF-IDF along with Weighted Frequency for Text Summarization?

I've been working on a text summarization problem. After studying this blog, I used weighted-frequencies of words present in a document for summarization. I would compute the weighted frequency ...
0
votes
1answer
13 views

How to identify new job descriptions/postings from a set of documents when I have a set of already labeled job descriptions/postings

Suppose I have a set of already labeled documents -- some of them are job descriptions/postings (these are documents of interest), and some of them are not. I wonder what kind of method would allow me ...
13
votes
3answers
16k views

What is the difference between a hashing vectorizer and a tfidf vectorizer

I'm converting a corpus of text documents into word vectors for each document. I've tried this using a TfidfVectorizer and a HashingVectorizer I understand that a ...
2
votes
1answer
13 views

Doubt on scope of text classification problem

I have a dataset that describes the sellers who are selling various brands. I need to identify the source (where did he buy those brands he is selling from) of those sellers. (Dimension of dataset 11,...
0
votes
0answers
15 views

How do I discern document structure from differently-tagged XML documents?

I have a body of PDF documents of differing vintage. Our group had exported the documents as text to feed them into a natural-language parser (I think) to pull out subject-verb-predicate triples. ...
3
votes
2answers
5k views

What are useful evaluation metrics used in machine learning

I am using CNN in order to predict codes after analyzing text. As an example, I will write "I am crazy" .. the model will predict some code " X321". All this based on CNN. I want to evaluate my ...
1
vote
1answer
40 views

Build text complexity model based on complex examples

I try to build the user specific model which predicts whether arbitrary English text is complex for particular user or not. Having the complex and easy text samples allows to build such model but what ...
0
votes
0answers
8 views

Looking for suggestions on performing Sementic Analysis of ASR text

Currently I am working on a project where I have ASR on which I am performing semantic analysis to extract meaning out of it. The ASR text contains huge amount of vague conversational text which needs ...
1
vote
1answer
42 views

Emotional tension score in sentences

I am beginner in natural language processing and my goal is to find a way to score sentences based on their emotional tension. More specifically, I would like to know to what degree a sentence ...
1
vote
1answer
27 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
1
vote
2answers
748 views

Getting uniform distribution over topics from gensim's LDA?

I am trying to learn topics distribution for each document in a corpus. I have term-document matrix (sparse matrix of dim: num_terms * no_docs) as input to the LDA model (with num_topics=100) and ...

1
2 3 4 5
11