Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
0
votes
2answers
19 views

Extract amount from free text

I want to extract various amounts and tenure of contracts from different contract documents that we have. For example : Mr xyz, this contact is valid for 3 Months and you have to pay $3000 as ...
0
votes
0answers
14 views

Get row wise frequency count of words from list in text column pandas

I have a data frame with a Audio Transcript column from customer care phone conversation. I have created one list with words and sentences ...
1
vote
1answer
20 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
-2
votes
0answers
12 views

How can we calculate the Semantic Textual Similarity between two paragraphs? [closed]

The dataset has 3 columns: Unique_ID, text1, text2. I have to calculate the Semantic Textual Similarity score between the text1 and text2 columns I am a novice to NLP. So, please provide the answer ...
1
vote
0answers
17 views

What is the state-of-the-art method/algorithm to extract Keywords from text?

What is the state-of-the-art method/algorithm to extract data from text without regarding the language of the text? The methods I am reading about are Rake, Yake or using LTSM Networks to identify ...
2
votes
1answer
41 views

NER vs Text classification for very short sentences

Given a large set of short sentences (around 20-30 words) and multi label task (around 100 labels , can be to 3 labels per sentences ). The location of each annotation is not impotent (i.e i only ...
0
votes
1answer
17 views

What does online learning mean in Topic modeling (LDA) - Gensim

I came across this line in the Gensim Documentation- Gensim LDA - "The model can also be updated with new documents for online training." So my assumption on what it means is - 'Once we have a ...
0
votes
0answers
9 views

Association Rule Mining across two market baskets

I am quite familiar with Association Rule mining but I need to use it to associate ACROSS two market baskets instead of finding support WITHIN a market basket. Imagine customers come to a Store A ...
1
vote
2answers
35 views

Reaching 100% accurray in Data Mining

I am currently working with Topic Models, especially LDA, and now I am asking myself if it's possible to reach total accurracy regarding the results. If I insepct the results of my Topic Model, the ...
0
votes
1answer
47 views

Word2Vec and Tf-idf how to combine them

I'm currently working in text mining ptoject I'd like to know once I'm on vectorisation. With method is better. Is it Word2Vec or ...
4
votes
1answer
95 views

NLP - paraphrase extraction in python

I am trying to develop a NLP model, which takes something like you have high levels of cholesterol(this will be a tag) as input and has to output something like <...
2
votes
0answers
16 views

Determining the ideal parameters for KeyGraph Algorithm

Is there a method to determine the ideal amount of Keywords, High Frequency Terms and High Key Terms for the Keygraph? To determine the ideal amount of topics for the LDA you have Topic Coherence or ...
0
votes
0answers
18 views

Encode features for Machine Learning Model

I am working on a classification problem on medical reports. I am taking ngrams as features. The problem is that there are few attributes that a single ngram can posses. For example, if 'abdominal ...
2
votes
2answers
26 views

Question answering (QA) vs Chatbots

Are Question answering (QA) the same as Chatbots? I can not understand the difference between them. For me it's the same thing: interact with a robot that answers questions.
1
vote
0answers
11 views

Extract editing history from Microsoft Word documents? [closed]

Is there a tool to computationally extract the editing history of a given Microsoft Word Document? I have been using Apache Tika, but can only extract the last version of the text, and meta-...
2
votes
3answers
64 views

How to approach TF-IDf based analysis?

Problem statement : We have documents with list of words in them. Overall these documents are classified into 2 group (say, good quality vs bad) docs - ...
2
votes
2answers
36 views

Which kind of model is better for keyword-set classification?

There exists a similar task that is named text classification. But I want to find a kind of model that the inputs are keyword set. And the keyword set is not from a sentence. For example: ...
5
votes
4answers
744 views

How to deal with spelling errors NLP

I have some data where the main column is the description of one product. The main task is to extract the name of some product from this column, where it sometimes is spelled wrong and amended in ...
0
votes
2answers
54 views

How to get a similarity vector from two vectors?

I want to make a classification model for 3 classes, i have 2 sentences for each observation, firstly i apply a cnn layer for each sentence and then i added dense layer. ...
0
votes
1answer
26 views

CV(Curriculum vitae) Recommendation System guidance

I am building a recommender system which matches people's CV with a vacancy. So far, I used TF-IDF & Cosine Similarity to get a matching score between a vacancy and a candidate's CV. I want to ...
1
vote
2answers
46 views

Machine Learning Analysis for Redaction Purposes of Personally Identifying Information from Open Text Fields

Let's say that I wanted to use machine learning to find and redact personally identifying information (PII) from millions of records with open text fields. Let's also say the PII could include a ...
0
votes
1answer
22 views

NLP: Getting the top 5 or top 10 predictions

I am working on a social networking application and I have to make its news feed better. For example: If someone searches for 'suggest me some good books', it should yield some names. Now, I have ...
0
votes
0answers
6 views

Find a specific paper on information retrieval method supporting literature research searching not by keywords but by documents

About 2012, I did a literature research on the following topic but unfortunately lost my results. Specifically one paper comes repeatedly to my mind, therefore maybe someone knows about this or ...
1
vote
1answer
182 views

Datasets for Topic Modeling [closed]

I'm looking to try and use deep learning methods for topic modeling as opposed to the more traditional methods of lda and word embedding methods. However, I'm having trouble finding good labeled ...
0
votes
0answers
12 views

Conceptual help generating a text adventure game using a GAN

I have built a playable dungeon crawler game that lets a character progress through a series of randomly generated rooms filled with doors, chests, stairs, etc. Ideally, I would be able to display a ...
2
votes
1answer
22 views

Attitude to text mining and preparing tokens, irrelevant words, low accuracy

For purpose of quite big project I am doing a text mining on some documents. My steps are quite common: All to lower case Tokenization Stop list and stop words Lemmatizaton Stemming Some other ...
0
votes
1answer
28 views

Naive Bayes / SVM classifiation - min. number of records (Python)

I am doing text classification with Python. I have around 120 records with 2 columns: text class I tokenize, stem and lematize the words, I also did some of my own text preprocessing. When I run the ...
1
vote
0answers
13 views

Populating Knowledge Base - Stanford DeepMind Alternatives

I am dealing with the task to extract structured information from domain-specific unstructured documents. The end goal is to obtain a reliable, queryable system, i.e. in the form of a chat-bot or ...
1
vote
2answers
41 views

Clustering Small Text Descriptions

Im presented with a unique text classification problem. Im given a list of descriptions each containing 3-8 words. I know that there are some descriptions that are nearly the same, but the majority ...
1
vote
2answers
29 views

How can you make use of Json format data?

I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an ...
0
votes
0answers
87 views

Unsupervised learning, Python, text clustering

I want to do unsupervised learning. As I understand with such learning we don't know the clusters before, right? I read about k-means alghoritm, followed mainly these two articles: The first article. ...
1
vote
0answers
21 views

PDF/Text to csv table

I have very little python experience but I have been wanting to get into data science and thought I would start with pdf/text mining. Because it's something I need right now anyway. I have a list of ...
0
votes
0answers
13 views

Looking for recommendations on auto-tagging approaches

Basically looking for approaches to solve the stackoverflow question tagging problem. I have seen a few papers already but in case I have missed something - asking here too. For anyone interested, ...
1
vote
0answers
26 views

Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?

I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
0
votes
0answers
24 views

Latent Dirichlet Allocation in R, topicmodels using VEM algorithm or Gibbs Sampling mixing tm and topicmodels library or WarpLDA from text2vec?

If I am trying to classify 230k text abstracts, which option would be better and more precise when aplying LDA?
3
votes
2answers
29 views

Classifying dates in sentences

Let's say I have a sentences that goes like this: Hi, how are the kids. I will be going to Los Angeles next Friday and will come back the following Monday. If the date today is October 16 (...
0
votes
1answer
32 views

Information Extraction/Semantic Search for long, unstructured documents

I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store ...
0
votes
0answers
10 views

How to approach the topical text categorization of a small collection of short texts?

I have a set of 200 very short documents, between 1 and 20 words each. One of my colleague would like to classify each of these documents in three predefined topics (let's call them "A", "B", and "C"...
0
votes
0answers
30 views

Collecting structured data from HTML source code: A generalized way

I am working on a task to build a generic function to extract some specific fields from HTML source code. The fields we want are such as product title, price, quantity and shipment The generic ...
0
votes
1answer
82 views

Finding cosine similarity score

I have a dataframe that looks like this: sentence intent hi greeting hello greeting buy this buy whats up conversation . . What I'd like ...
0
votes
1answer
30 views

Why is TF IDF output lognormal?

I ran a TF IDF algorithm and the result of predicted similarities using cosine similarity is a log-normal distribution. Is this a feature of the algorithm (e.g., all logit probabilities are log-...
0
votes
0answers
15 views

POS extraction using CoreNLP

I have a corpus of windows related documents, for which I need to extract nouns and verbs. However, it is required that I keep certain windows specific words such as "inline hooking", "instruction ...
1
vote
1answer
32 views

Can you suggest a better to organise the data to generate frequent itemsets?

I have a data of a bag of words in a document. The data has 3 coulumns : document number, word number, count of the word in the number. I am supposed to generate frequent itemsets of particular size. ...
-1
votes
1answer
46 views

appending file text into list with file name in python [closed]

Here 20_newsgroups is a folder which contain 20 folder and each folder contain some file , i just want to make datasets containind words and file name its gives Error ...
1
vote
1answer
18 views

Infer family type, size from reviews

I have a bunch of reviews: ...
0
votes
0answers
21 views

Probabilistic model of selecting subsets of words from documents?

Is there an existing probabilistic model that deals with the selection of subsets of words from a corpus of documents? Imagine a stack of documents where a subset of the words in each document has ...
0
votes
0answers
62 views

how to calculate coherence score in topic model

I am trying to calculate coherence score in topic modeling. I am following this Github link So there I need to use the preprocessed wiki and news. I got 3 questions: if the domain that I have ...
0
votes
0answers
15 views

Can we treat sentiment score of the review text as rating of the product?

I have a review text of the different products, And I need the rating of the product. So can we use sentiment score as the rating of the product.
0
votes
0answers
11 views

Is it possible to build an intelligent lead classifier with just a few training units

I want to build a lead classifier for my Master Thesis and wanted to ask for an assessment of feasibility. Here are the key points: (1) We have 15 customers and about 100 opportunities of which we ...
0
votes
2answers
191 views

NLP to detect duplicates for very technical language

I have the following scenario, to detect duplicate products based on the description fields. The Description Field contains product technical name, dimensions, characteristics. My model needs to ...

1 2 3 4 5 10