Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
1
vote
1answer
14 views

Which insights a data scientist could derive from text-analysis?

I have many texts and I am trying to analyse them. After tokenising them, studying words frequency, spotting any typos, studying punctuations, I have been working on POS tagging. Since it is my first ...
1
vote
2answers
37 views

Extract information using NLP and store it in csv file

I have a text file that stores the pickup, drops, and time. SMS text is a dummy file that is used to train a cab service model. The text is like in this format: ...
1
vote
1answer
27 views

Extract First names from usernames

John10 , michaelscott, James.white , Jr.Jones , James-Anderson ,WhiteWalter10 -- These are some of the different cases of usernames possible(there may be more ). I have about 200K such usernames . I ...
0
votes
0answers
4 views

Dirichlet smoothing as an IDF component

How can Dirichlet smoothing be used as an IDF component to estimate the probabilities of a Topic model ? i.e, Smoothing with a background collection model to estimate topic model ? I've seen many ...
0
votes
1answer
19 views

Why do i get error when I use corpus in Orange v 3.25.0 text mining widget and no import document option?

I am experimenting with Orange data mining tool. When i use 'Corpus' from text mining. It gives me error. . I tried many things, but still unable to resolve this issue. Besides that, In text mining, ...
0
votes
0answers
21 views

How to segregate resume layouts into different types?

I'm looking for any suggestions on how to segregate resume layout into different types. How do one proceed with such a task? I mean resumes are usually available as pdf or docx format and when we ...
0
votes
1answer
24 views

breaking joined words into meaningful ones during text mining

I'm performing an aspect-based sentiment on consumer complaints. I'm tokenizing at the sentence level. ...
1
vote
1answer
10 views

How to represent a document in test data with the Document-Term Matrix created from the training set?

I build a classifier of documents using the vector representation of each document in the training set (i.e a row in the Document-Term Matrix). Now I need to test the model on the test data. But how ...
0
votes
0answers
11 views

How to build a resume text block segmentation algorithm using deep learning?

Assume I have resume, I want to segment different text blocks such as personal info, education , experience etc.., Is is possible to segment text blocks by converting pdf or docx resume into image ...
0
votes
0answers
17 views

English Word popularity scoring

I'm new to data science and looking for the word popularity ranking algorithm. Given a list of words, What would the best and easy to implement algorithm to score each word based on the word ...
0
votes
1answer
20 views

Text Mapping - Medicine Names

We have a problem where we have a standardized database of Medicine names. On the other hand, there is a subset of medicine names which could have spelling mistakes, different structure or hypens, ...
0
votes
0answers
9 views

How to identify corresponding record of a car from a semi-structured string?

I am trying to build an application that can take a record of a car from different websites, compare it to data i have in a CSV file and return me the matching row. Each website will present and ...
0
votes
0answers
8 views

What type of deep learning model should I use to extract fields from legal agreement documents?

I have rent agreement documents and a csv file having output column fields (labels) like agreement amount,name of tenant etc. For every document an AI model should ...
0
votes
0answers
14 views

Benchmark queries for Benchmark dataset with ranked list of documents

I aim to evaluate the ranking of an information-retrieval system. For a benchmark dataset like TREC, I have followed the qrels file which has list of documents for a particular topic (query) with ...
0
votes
1answer
20 views

Get a prediction from the new data inputted against the model, but an error is produced, how to adapt the R code for it to work?

In the R code below, I included the sentences when looking to compare the manually classified with lexicon dictionary results by positive, negative and neutral (in matrixdata1), the algorithms results ...
0
votes
0answers
34 views

How do I extract album and song titles from this plain text file?

Inspired by topic modeling and clustering analysis of Taylor Swift's lyrics, I want to do the same for the band Nightwish. I scraped Dark Lyrics (see script) for all of their lyrics and saved the ...
0
votes
0answers
4 views

Can text analysis approaches using machine learning be used on financial statement reports?

In a paper (Correa et al, 2017), the following is mentioned: " Alternative text analysis methods, such as machine-learning approaches, require some type of classification that would help in ...
0
votes
0answers
11 views

Named Entity Recognition/Linking with a dictionary of names (e.g., move titles, book titles, company names, people names)

I have a lot of text messages on which I want to perform Named Entity Recognition and Named Entity Linking. For example, I want to detect all movie names and find their IMDB ID. To make things easier,...
0
votes
0answers
7 views

Merging (intersecting) more than two posting list in linear time

The intersecting algorithm for two posting lists implemented below: ...
0
votes
1answer
38 views

Does Python have R's tidytext equivalent?

I can't seem to find a tidytext (R library) equivalent in Python. Text mining in Python seems quite weak compared to R.
0
votes
0answers
10 views

Can we use TF-IDF along with Weighted Frequency for Text Summarization?

I've been working on a text summarization problem. After studying this blog, I used weighted-frequencies of words present in a document for summarization. I would compute the weighted frequency ...
0
votes
0answers
15 views

How do I discern document structure from differently-tagged XML documents?

I have a body of PDF documents of differing vintage. Our group had exported the documents as text to feed them into a natural-language parser (I think) to pull out subject-verb-predicate triples. ...
0
votes
1answer
50 views

NLP - Simple approach to identify commonalities in text comments between people

For something we are working on, we were looking for a simple way to compare from review/feedback data against a question (for which there are multiple responses from multiple people), the following: ...
0
votes
0answers
8 views

Looking for suggestions on performing Sementic Analysis of ASR text

Currently I am working on a project where I have ASR on which I am performing semantic analysis to extract meaning out of it. The ASR text contains huge amount of vague conversational text which needs ...
0
votes
1answer
13 views

How to identify new job descriptions/postings from a set of documents when I have a set of already labeled job descriptions/postings

Suppose I have a set of already labeled documents -- some of them are job descriptions/postings (these are documents of interest), and some of them are not. I wonder what kind of method would allow me ...
2
votes
1answer
13 views

Doubt on scope of text classification problem

I have a dataset that describes the sellers who are selling various brands. I need to identify the source (where did he buy those brands he is selling from) of those sellers. (Dimension of dataset 11,...
0
votes
0answers
18 views

Hazm: POSTagger(): ArgumentError: argument 2: <class 'TypeError'>: wrong type

I have got an error for running the below code. May you give me some help? ...
1
vote
1answer
15 views

how to create a searchable tree on Persian text?

I wanna clean a Persian text from stop-words. I already have stop-word data that is provided on the below link. It seems to me, if I have a pre-built tree on stop-words, I could save lots of time. I ...
0
votes
2answers
29 views

Extract amount from free text

I want to extract various amounts and tenure of contracts from different contract documents that we have. For example : Mr xyz, this contact is valid for 3 Months and you have to pay $3000 as ...
0
votes
0answers
21 views

Get row wise frequency count of words from list in text column pandas

I have a data frame with a Audio Transcript column from customer care phone conversation. I have created one list with words and sentences ...
1
vote
1answer
27 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
1
vote
0answers
21 views

What is the state-of-the-art method/algorithm to extract Keywords from text?

What is the state-of-the-art method/algorithm to extract data from text without regarding the language of the text? The methods I am reading about are Rake, Yake or using LTSM Networks to identify ...
2
votes
1answer
108 views

NER vs Text classification for very short sentences

Given a large set of short sentences (around 20-30 words) and multi label task (around 100 labels , can be to 3 labels per sentences ). The location of each annotation is not impotent (i.e i only ...
0
votes
1answer
25 views

What does online learning mean in Topic modeling (LDA) - Gensim

I came across this line in the Gensim Documentation- Gensim LDA - "The model can also be updated with new documents for online training." So my assumption on what it means is - 'Once we have a ...
0
votes
1answer
25 views

Association Rule Mining across two market baskets

I am quite familiar with Association Rule mining but I need to use it to associate ACROSS two market baskets instead of finding support WITHIN a market basket. Imagine customers come to a Store A ...
1
vote
2answers
38 views

Reaching 100% accurray in Data Mining

I am currently working with Topic Models, especially LDA, and now I am asking myself if it's possible to reach total accurracy regarding the results. If I insepct the results of my Topic Model, the ...
1
vote
1answer
576 views

Word2Vec and Tf-idf how to combine them

I'm currently working in text mining ptoject I'd like to know once I'm on vectorisation. With method is better. Is it Word2Vec or ...
4
votes
1answer
104 views

NLP - paraphrase extraction in python

I am trying to develop a NLP model, which takes something like you have high levels of cholesterol(this will be a tag) as input and has to output something like <...
2
votes
0answers
16 views

Determining the ideal parameters for KeyGraph Algorithm

Is there a method to determine the ideal amount of Keywords, High Frequency Terms and High Key Terms for the Keygraph? To determine the ideal amount of topics for the LDA you have Topic Coherence or ...
0
votes
0answers
21 views

Encode features for Machine Learning Model

I am working on a classification problem on medical reports. I am taking ngrams as features. The problem is that there are few attributes that a single ngram can posses. For example, if 'abdominal ...
2
votes
2answers
118 views

Question answering (QA) vs Chatbots

Are Question answering (QA) the same as Chatbots? I can not understand the difference between them. For me it's the same thing: interact with a robot that answers questions.
1
vote
0answers
16 views

Extract editing history from Microsoft Word documents? [closed]

Is there a tool to computationally extract the editing history of a given Microsoft Word Document? I have been using Apache Tika, but can only extract the last version of the text, and meta-...
2
votes
3answers
71 views

How to approach TF-IDf based analysis?

Problem statement : We have documents with list of words in them. Overall these documents are classified into 2 group (say, good quality vs bad) docs - ...
2
votes
2answers
42 views

Which kind of model is better for keyword-set classification?

There exists a similar task that is named text classification. But I want to find a kind of model that the inputs are keyword set. And the keyword set is not from a sentence. For example: ...
6
votes
4answers
852 views

How to deal with spelling errors NLP

I have some data where the main column is the description of one product. The main task is to extract the name of some product from this column, where it sometimes is spelled wrong and amended in ...
0
votes
2answers
60 views

How to get a similarity vector from two vectors?

I want to make a classification model for 3 classes, i have 2 sentences for each observation, firstly i apply a cnn layer for each sentence and then i added dense layer. ...
0
votes
1answer
31 views

CV(Curriculum vitae) Recommendation System guidance

I am building a recommender system which matches people's CV with a vacancy. So far, I used TF-IDF & Cosine Similarity to get a matching score between a vacancy and a candidate's CV. I want to ...
1
vote
2answers
89 views

Machine Learning Analysis for Redaction Purposes of Personally Identifying Information from Open Text Fields

Let's say that I wanted to use machine learning to find and redact personally identifying information (PII) from millions of records with open text fields. Let's also say the PII could include a ...
0
votes
2answers
29 views

NLP: Getting the top 5 or top 10 predictions

I am working on a social networking application and I have to make its news feed better. For example: If someone searches for 'suggest me some good books', it should yield some names. Now, I have ...
0
votes
0answers
6 views

Find a specific paper on information retrieval method supporting literature research searching not by keywords but by documents

About 2012, I did a literature research on the following topic but unfortunately lost my results. Specifically one paper comes repeatedly to my mind, therefore maybe someone knows about this or ...

1
2 3 4 5
10