Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
0answers
94 views

How do I extract album and song titles from this plain text file?

Inspired by topic modeling and clustering analysis of Taylor Swift's lyrics, I want to do the same for the band Nightwish. I scraped Dark Lyrics (see script) for all of their lyrics and saved the ...
1
vote
0answers
70 views

Merging (intersecting) more than two posting list in linear time

The intersecting algorithm for two posting lists implemented below: ...
1
vote
2answers
618 views

Does Python have R's tidytext equivalent?

I can't seem to find a tidytext (R library) equivalent in Python. Text mining in Python seems quite weak compared to R.
1
vote
0answers
22 views

How do I discern document structure from differently-tagged XML documents?

I have a body of PDF documents of differing vintage. Our group had exported the documents as text to feed them into a natural-language parser (I think) to pull out subject-verb-predicate triples. ...
1
vote
1answer
112 views

NLP - Simple approach to identify commonalities in text comments between people

For something we are working on, we were looking for a simple way to compare from review/feedback data against a question (for which there are multiple responses from multiple people), the following: ...
1
vote
0answers
10 views

Looking for suggestions on performing Sementic Analysis of ASR text

Currently I am working on a project where I have ASR on which I am performing semantic analysis to extract meaning out of it. The ASR text contains huge amount of vague conversational text which needs ...
0
votes
1answer
21 views

How to identify new job descriptions/postings from a set of documents when I have a set of already labeled job descriptions/postings

Suppose I have a set of already labeled documents -- some of them are job descriptions/postings (these are documents of interest), and some of them are not. I wonder what kind of method would allow me ...
3
votes
1answer
21 views

Doubt on scope of text classification problem

I have a dataset that describes the sellers who are selling various brands. I need to identify the source (where did he buy those brands he is selling from) of those sellers. (Dimension of dataset 11,...
1
vote
1answer
20 views

how to create a searchable tree on Persian text?

I wanna clean my huge text data from stop-words. I already have stop-word data that is provided on the below link. It seems to me, if I have a pre-built tree on stop-words, I could save lots of time. ...
0
votes
2answers
118 views

Extracting amount from free text

I want to extract various amounts and tenure of contracts from different contract documents that we have. For example: Mr xyz, this contact is valid for 3 Months and you have to pay $3000 as agreement ...
2
votes
1answer
384 views

Get row wise frequency count of words from list in text column pandas

I have a data frame with a Audio Transcript column from customer care phone conversation. I have created one list with words and sentences ...
1
vote
1answer
59 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
1
vote
0answers
50 views

What is the state-of-the-art method/algorithm to extract Keywords from text?

What is the state-of-the-art method/algorithm to extract data from text without regarding the language of the text? The methods I am reading about are Rake, Yake or using LTSM Networks to identify ...
2
votes
1answer
623 views

NER vs Text classification for very short sentences

Given a large set of short sentences (around 20-30 words) and multi label task (around 100 labels , can be to 3 labels per sentences ). The location of each annotation is not impotent (i.e i only ...
1
vote
1answer
309 views

What does online learning mean in Topic modeling (LDA) - Gensim

I came across this line in the Gensim Documentation- Gensim LDA - "The model can also be updated with new documents for online training." So my assumption on what it means is - 'Once we have a ...
1
vote
1answer
49 views

Association Rule Mining across two market baskets

I am quite familiar with Association Rule mining but I need to use it to associate ACROSS two market baskets instead of finding support WITHIN a market basket. Imagine customers come to a Store A ...
1
vote
2answers
43 views

Reaching 100% accurray in Data Mining

I am currently working with Topic Models, especially LDA, and now I am asking myself if it's possible to reach total accurracy regarding the results. If I insepct the results of my Topic Model, the ...
3
votes
1answer
4k views

Word2Vec and Tf-idf how to combine them

I'm currently working in text mining ptoject I'd like to know once I'm on vectorisation. With method is better. Is it Word2Vec or ...
4
votes
1answer
253 views

NLP - paraphrase extraction in python

I am trying to develop a NLP model, which takes something like you have high levels of cholesterol(this will be a tag) as input and has to output something like <...
2
votes
0answers
20 views

Determining the ideal parameters for KeyGraph Algorithm

Is there a method to determine the ideal amount of Keywords, High Frequency Terms and High Key Terms for the Keygraph? To determine the ideal amount of topics for the LDA you have Topic Coherence or ...
2
votes
2answers
1k views

Question answering (QA) vs Chatbots

Are Question answering (QA) the same as Chatbots? I can not understand the difference between them. For me it's the same thing: interact with a robot that answers questions.
1
vote
0answers
130 views

Extract editing history from Microsoft Word documents? [closed]

Is there a tool to computationally extract the editing history of a given Microsoft Word Document? I have been using Apache Tika, but can only extract the last version of the text, and meta-...
2
votes
3answers
74 views

How to approach TF-IDf based analysis?

Problem statement : We have documents with list of words in them. Overall these documents are classified into 2 group (say, good quality vs bad) docs - ...
2
votes
2answers
52 views

Which kind of model is better for keyword-set classification?

There exists a similar task that is named text classification. But I want to find a kind of model that the inputs are keyword set. And the keyword set is not from a sentence. For example: ...
8
votes
4answers
3k views

How to deal with spelling errors NLP

I have some data where the main column is the description of one product. The main task is to extract the name of some product from this column, where it sometimes is spelled wrong and amended in ...
0
votes
2answers
85 views

How to get a similarity vector from two vectors?

I want to make a classification model for 3 classes, i have 2 sentences for each observation, firstly i apply a cnn layer for each sentence and then i added dense layer. ...
0
votes
1answer
53 views

CV(Curriculum vitae) Recommendation System guidance

I am building a recommender system which matches people's CV with a vacancy. So far, I used TF-IDF & Cosine Similarity to get a matching score between a vacancy and a candidate's CV. I want to ...
1
vote
2answers
215 views

Machine Learning Analysis for Redaction Purposes of Personally Identifying Information from Open Text Fields

Let's say that I wanted to use machine learning to find and redact personally identifying information (PII) from millions of records with open text fields. Let's also say the PII could include a ...
0
votes
2answers
40 views

NLP: Getting the top 5 or top 10 predictions

I am working on a social networking application and I have to make its news feed better. For example: If someone searches for 'suggest me some good books', it should yield some names. Now, I have ...
1
vote
1answer
2k views

Datasets for Topic Modeling [closed]

I'm looking to try and use deep learning methods for topic modeling as opposed to the more traditional methods of lda and word embedding methods. However, I'm having trouble finding good labeled ...
2
votes
1answer
41 views

Attitude to text mining and preparing tokens, irrelevant words, low accuracy

For purpose of quite big project I am doing a text mining on some documents. My steps are quite common: All to lower case Tokenization Stop list and stop words Lemmatizaton Stemming Some other ...
0
votes
1answer
31 views

Naive Bayes / SVM classifiation - min. number of records (Python)

I am doing text classification with Python. I have around 120 records with 2 columns: text class I tokenize, stem and lematize the words, I also did some of my own text preprocessing. When I run the ...
1
vote
0answers
19 views

Populating Knowledge Base - Stanford DeepMind Alternatives

I am dealing with the task to extract structured information from domain-specific unstructured documents. The end goal is to obtain a reliable, queryable system, i.e. in the form of a chat-bot or ...
1
vote
2answers
142 views

Clustering Small Text Descriptions

Im presented with a unique text classification problem. Im given a list of descriptions each containing 3-8 words. I know that there are some descriptions that are nearly the same, but the majority ...
1
vote
2answers
39 views

How can you make use of Json format data?

I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an ...
2
votes
0answers
66 views

Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?

I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
1
vote
0answers
152 views

Latent Dirichlet Allocation in R, topicmodels using VEM algorithm or Gibbs Sampling mixing tm and topicmodels library or WarpLDA from text2vec?

If I am trying to classify 230k text abstracts, which option would be better and more precise when aplying LDA?
3
votes
2answers
40 views

Classifying dates in sentences

Let's say I have a sentences that goes like this: Hi, how are the kids. I will be going to Los Angeles next Friday and will come back the following Monday. If the date today is October 16 (...
1
vote
1answer
107 views

Information Extraction/Semantic Search for long, unstructured documents

I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store ...
-1
votes
1answer
480 views

Finding cosine similarity score

I have a dataframe that looks like this: sentence intent hi greeting hello greeting buy this buy whats up conversation . . What I'd like ...
0
votes
1answer
71 views

Why is TF IDF output lognormal?

I ran a TF IDF algorithm and the result of predicted similarities using cosine similarity is a log-normal distribution. Is this a feature of the algorithm (e.g., all logit probabilities are log-...
1
vote
1answer
51 views

Suggestion for a better way to organize data to generate frequent item-sets?

I have a data of a bag of words in a document. The data has 3 columns: {document number, word number, count of the word in the number}. I am supposed to generate ...
1
vote
1answer
18 views

Infer family type, size from reviews

I have a bunch of reviews: ...
1
vote
3answers
1k views

NLP to detect duplicates for very technical language

I have the following scenario, to detect duplicate products based on the description fields. The Description Field contains product technical name, dimensions, characteristics. My model needs to ...
1
vote
0answers
61 views

Sentiment Analysis for Q&A based reviews

I'm a self-learning ML enthusiast and I recently started learning NLP and performing Sentiment Analysis on imdb, yelp, amazon datasets(using Python). I came across a dataset where the reviews were in ...
1
vote
1answer
39 views

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them?

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them? That is, to get results which say that, for example, the ...
0
votes
1answer
43 views

Which are the appropriate prameters for lda modeling?

I try to implement in R test for appropriate metrics for lda. Here the way I try to use LDA ...
2
votes
0answers
49 views

Text extraction / mining from specific templates (ML)

believe me if I say that I have read basically all the threads in this website regarding this subject. A lot of them have a similar title, but the problem is somehow different. Small context: just ...
3
votes
1answer
49 views

How to segment old digitized newspapers into articles

I'm working on a large corpus of french daily newspapers from the 19th century that have been digitized and where the data are in the form of raw OCR text files (one text file per day). In terms of ...
1
vote
1answer
552 views

Sentiment Analysis of News Headlines

I'm trying to do sentiment analysis of News Headlines about a particular subject mentioned in it. Initially, I used TextBlob library for sentiment analysis to ...

1 2
3
4 5
12