Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

Looking for suggestions on performing Sementic Analysis of ASR text

Currently I am working on a project where I have ASR on which I am performing semantic analysis to extract meaning out of it. The ASR text contains huge amount of vague conversational text which needs ...
0
votes
1answer
11 views

How to identify new job descriptions/postings from a set of documents when I have a set of already labeled job descriptions/postings

Suppose I have a set of already labeled documents -- some of them are job descriptions/postings (these are documents of interest), and some of them are not. I wonder what kind of method would allow me ...
2
votes
1answer
10 views

Doubt on scope of text classification problem

I have a dataset that describes the sellers who are selling various brands. I need to identify the source (where did he buy those brands he is selling from) of those sellers. (Dimension of dataset 11,...
0
votes
0answers
16 views

Hazm: POSTagger(): ArgumentError: argument 2: <class 'TypeError'>: wrong type

I have got an error for running the below code. May you give me some help? ...
1
vote
1answer
15 views

how to create a searchable tree on Persian text?

I wanna clean a Persian text from stop-words. I already have stop-word data that is provided on the below link. It seems to me, if I have a pre-built tree on stop-words, I could save lots of time. I ...
0
votes
2answers
23 views

Extract amount from free text

I want to extract various amounts and tenure of contracts from different contract documents that we have. For example : Mr xyz, this contact is valid for 3 Months and you have to pay $3000 as ...
0
votes
0answers
16 views

Get row wise frequency count of words from list in text column pandas

I have a data frame with a Audio Transcript column from customer care phone conversation. I have created one list with words and sentences ...
1
vote
1answer
24 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
1
vote
0answers
18 views

What is the state-of-the-art method/algorithm to extract Keywords from text?

What is the state-of-the-art method/algorithm to extract data from text without regarding the language of the text? The methods I am reading about are Rake, Yake or using LTSM Networks to identify ...
2
votes
1answer
63 views

NER vs Text classification for very short sentences

Given a large set of short sentences (around 20-30 words) and multi label task (around 100 labels , can be to 3 labels per sentences ). The location of each annotation is not impotent (i.e i only ...
0
votes
1answer
22 views

What does online learning mean in Topic modeling (LDA) - Gensim

I came across this line in the Gensim Documentation- Gensim LDA - "The model can also be updated with new documents for online training." So my assumption on what it means is - 'Once we have a ...
0
votes
1answer
18 views

Association Rule Mining across two market baskets

I am quite familiar with Association Rule mining but I need to use it to associate ACROSS two market baskets instead of finding support WITHIN a market basket. Imagine customers come to a Store A ...
1
vote
2answers
37 views

Reaching 100% accurray in Data Mining

I am currently working with Topic Models, especially LDA, and now I am asking myself if it's possible to reach total accurracy regarding the results. If I insepct the results of my Topic Model, the ...
0
votes
1answer
150 views

Word2Vec and Tf-idf how to combine them

I'm currently working in text mining ptoject I'd like to know once I'm on vectorisation. With method is better. Is it Word2Vec or ...
4
votes
1answer
102 views

NLP - paraphrase extraction in python

I am trying to develop a NLP model, which takes something like you have high levels of cholesterol(this will be a tag) as input and has to output something like <...
2
votes
0answers
16 views

Determining the ideal parameters for KeyGraph Algorithm

Is there a method to determine the ideal amount of Keywords, High Frequency Terms and High Key Terms for the Keygraph? To determine the ideal amount of topics for the LDA you have Topic Coherence or ...
0
votes
0answers
18 views

Encode features for Machine Learning Model

I am working on a classification problem on medical reports. I am taking ngrams as features. The problem is that there are few attributes that a single ngram can posses. For example, if 'abdominal ...
2
votes
2answers
45 views

Question answering (QA) vs Chatbots

Are Question answering (QA) the same as Chatbots? I can not understand the difference between them. For me it's the same thing: interact with a robot that answers questions.
1
vote
0answers
13 views

Extract editing history from Microsoft Word documents? [closed]

Is there a tool to computationally extract the editing history of a given Microsoft Word Document? I have been using Apache Tika, but can only extract the last version of the text, and meta-...
2
votes
3answers
68 views

How to approach TF-IDf based analysis?

Problem statement : We have documents with list of words in them. Overall these documents are classified into 2 group (say, good quality vs bad) docs - ...
2
votes
2answers
37 views

Which kind of model is better for keyword-set classification?

There exists a similar task that is named text classification. But I want to find a kind of model that the inputs are keyword set. And the keyword set is not from a sentence. For example: ...
6
votes
4answers
777 views

How to deal with spelling errors NLP

I have some data where the main column is the description of one product. The main task is to extract the name of some product from this column, where it sometimes is spelled wrong and amended in ...
0
votes
2answers
57 views

How to get a similarity vector from two vectors?

I want to make a classification model for 3 classes, i have 2 sentences for each observation, firstly i apply a cnn layer for each sentence and then i added dense layer. ...
0
votes
1answer
26 views

CV(Curriculum vitae) Recommendation System guidance

I am building a recommender system which matches people's CV with a vacancy. So far, I used TF-IDF & Cosine Similarity to get a matching score between a vacancy and a candidate's CV. I want to ...
1
vote
2answers
63 views

Machine Learning Analysis for Redaction Purposes of Personally Identifying Information from Open Text Fields

Let's say that I wanted to use machine learning to find and redact personally identifying information (PII) from millions of records with open text fields. Let's also say the PII could include a ...
0
votes
1answer
23 views

NLP: Getting the top 5 or top 10 predictions

I am working on a social networking application and I have to make its news feed better. For example: If someone searches for 'suggest me some good books', it should yield some names. Now, I have ...
0
votes
0answers
6 views

Find a specific paper on information retrieval method supporting literature research searching not by keywords but by documents

About 2012, I did a literature research on the following topic but unfortunately lost my results. Specifically one paper comes repeatedly to my mind, therefore maybe someone knows about this or ...
1
vote
1answer
264 views

Datasets for Topic Modeling [closed]

I'm looking to try and use deep learning methods for topic modeling as opposed to the more traditional methods of lda and word embedding methods. However, I'm having trouble finding good labeled ...
0
votes
0answers
12 views

Conceptual help generating a text adventure game using a GAN

I have built a playable dungeon crawler game that lets a character progress through a series of randomly generated rooms filled with doors, chests, stairs, etc. Ideally, I would be able to display a ...
2
votes
1answer
24 views

Attitude to text mining and preparing tokens, irrelevant words, low accuracy

For purpose of quite big project I am doing a text mining on some documents. My steps are quite common: All to lower case Tokenization Stop list and stop words Lemmatizaton Stemming Some other ...
0
votes
1answer
29 views

Naive Bayes / SVM classifiation - min. number of records (Python)

I am doing text classification with Python. I have around 120 records with 2 columns: text class I tokenize, stem and lematize the words, I also did some of my own text preprocessing. When I run the ...
1
vote
0answers
13 views

Populating Knowledge Base - Stanford DeepMind Alternatives

I am dealing with the task to extract structured information from domain-specific unstructured documents. The end goal is to obtain a reliable, queryable system, i.e. in the form of a chat-bot or ...
1
vote
2answers
46 views

Clustering Small Text Descriptions

Im presented with a unique text classification problem. Im given a list of descriptions each containing 3-8 words. I know that there are some descriptions that are nearly the same, but the majority ...
1
vote
2answers
30 views

How can you make use of Json format data?

I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an ...
0
votes
0answers
93 views

Unsupervised learning, Python, text clustering

I want to do unsupervised learning. As I understand with such learning we don't know the clusters before, right? I read about k-means alghoritm, followed mainly these two articles: The first article. ...
1
vote
0answers
22 views

PDF/Text to csv table

I have very little python experience but I have been wanting to get into data science and thought I would start with pdf/text mining. Because it's something I need right now anyway. I have a list of ...
0
votes
0answers
13 views

Looking for recommendations on auto-tagging approaches

Basically looking for approaches to solve the stackoverflow question tagging problem. I have seen a few papers already but in case I have missed something - asking here too. For anyone interested, ...
1
vote
0answers
28 views

Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?

I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
0
votes
0answers
28 views

Latent Dirichlet Allocation in R, topicmodels using VEM algorithm or Gibbs Sampling mixing tm and topicmodels library or WarpLDA from text2vec?

If I am trying to classify 230k text abstracts, which option would be better and more precise when aplying LDA?
3
votes
2answers
30 views

Classifying dates in sentences

Let's say I have a sentences that goes like this: Hi, how are the kids. I will be going to Los Angeles next Friday and will come back the following Monday. If the date today is October 16 (...
0
votes
1answer
38 views

Information Extraction/Semantic Search for long, unstructured documents

I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store ...
0
votes
0answers
10 views

How to approach the topical text categorization of a small collection of short texts?

I have a set of 200 very short documents, between 1 and 20 words each. One of my colleague would like to classify each of these documents in three predefined topics (let's call them "A", "B", and "C"...
0
votes
0answers
30 views

Collecting structured data from HTML source code: A generalized way

I am working on a task to build a generic function to extract some specific fields from HTML source code. The fields we want are such as product title, price, quantity and shipment The generic ...
0
votes
1answer
149 views

Finding cosine similarity score

I have a dataframe that looks like this: sentence intent hi greeting hello greeting buy this buy whats up conversation . . What I'd like ...
0
votes
1answer
32 views

Why is TF IDF output lognormal?

I ran a TF IDF algorithm and the result of predicted similarities using cosine similarity is a log-normal distribution. Is this a feature of the algorithm (e.g., all logit probabilities are log-...
0
votes
0answers
17 views

POS extraction using CoreNLP

I have a corpus of windows related documents, for which I need to extract nouns and verbs. However, it is required that I keep certain windows specific words such as "inline hooking", "instruction ...
1
vote
1answer
33 views

Can you suggest a better to organise the data to generate frequent itemsets?

I have a data of a bag of words in a document. The data has 3 coulumns : document number, word number, count of the word in the number. I am supposed to generate frequent itemsets of particular size. ...
1
vote
1answer
18 views

Infer family type, size from reviews

I have a bunch of reviews: ...
0
votes
0answers
21 views

Probabilistic model of selecting subsets of words from documents?

Is there an existing probabilistic model that deals with the selection of subsets of words from a corpus of documents? Imagine a stack of documents where a subset of the words in each document has ...
0
votes
0answers
71 views

how to calculate coherence score in topic model

I am trying to calculate coherence score in topic modeling. I am following this Github link So there I need to use the preprocessed wiki and news. I got 3 questions: if the domain that I have ...

1
2 3 4 5
10