Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
2
votes
0answers
12 views

Determining the ideal parameters for KeyGraph Algorithm

Is there a method to determine the ideal amount of Keywords, High Frequency Terms and High Key Terms for the Keygraph? To determine the ideal amount of topics for the LDA you have Topic Coherence or ...
0
votes
0answers
18 views

Encode features for Machine Learning Model

I am working on a classification problem on medical reports. I am taking ngrams as features. The problem is that there are few attributes that a single ngram can posses. For example, if 'abdominal ...
2
votes
2answers
22 views

Question answering (QA) vs Chatbots

Are Question answering (QA) the same as Chatbots? I can not understand the difference between them. For me it's the same thing: interact with a robot that answers questions.
1
vote
0answers
11 views

Extract editing history from Microsoft Word documents? [closed]

Is there a tool to computationally extract the editing history of a given Microsoft Word Document? I have been using Apache Tika, but can only extract the last version of the text, and meta-...
2
votes
3answers
62 views

How to approach TF-IDf based analysis?

Problem statement : We have documents with list of words in them. Overall these documents are classified into 2 group (say, good quality vs bad) docs - ...
2
votes
2answers
35 views

Which kind of model is better for keyword-set classification?

There exists a similar task that is named text classification. But I want to find a kind of model that the inputs are keyword set. And the keyword set is not from a sentence. For example: ...
5
votes
4answers
719 views

How to deal with spelling errors NLP

I have some data where the main column is the description of one product. The main task is to extract the name of some product from this column, where it sometimes is spelled wrong and amended in ...
0
votes
2answers
50 views

How to get a similarity vector from two vectors?

I want to make a classification model for 3 classes, i have 2 sentences for each observation, firstly i apply a cnn layer for each sentence and then i added dense layer. ...
0
votes
1answer
24 views

CV(Curriculum vitae) Recommendation System guidance

I am building a recommender system which matches people's CV with a vacancy. So far, I used TF-IDF & Cosine Similarity to get a matching score between a vacancy and a candidate's CV. I want to ...
1
vote
2answers
35 views

Machine Learning Analysis for Redaction Purposes of Personally Identifying Information from Open Text Fields

Let's say that I wanted to use machine learning to find and redact personally identifying information (PII) from millions of records with open text fields. Let's also say the PII could include a ...
0
votes
1answer
22 views

NLP: Getting the top 5 or top 10 predictions

I am working on a social networking application and I have to make its news feed better. For example: If someone searches for 'suggest me some good books', it should yield some names. Now, I have ...
0
votes
0answers
6 views

Find a specific paper on information retrieval method supporting literature research searching not by keywords but by documents

About 2012, I did a literature research on the following topic but unfortunately lost my results. Specifically one paper comes repeatedly to my mind, therefore maybe someone knows about this or ...
1
vote
1answer
101 views

Datasets for Topic Modeling [closed]

I'm looking to try and use deep learning methods for topic modeling as opposed to the more traditional methods of lda and word embedding methods. However, I'm having trouble finding good labeled ...
0
votes
0answers
11 views

Conceptual help generating a text adventure game using a GAN

I have built a playable dungeon crawler game that lets a character progress through a series of randomly generated rooms filled with doors, chests, stairs, etc. Ideally, I would be able to display a ...
2
votes
1answer
22 views

Attitude to text mining and preparing tokens, irrelevant words, low accuracy

For purpose of quite big project I am doing a text mining on some documents. My steps are quite common: All to lower case Tokenization Stop list and stop words Lemmatizaton Stemming Some other ...
0
votes
1answer
28 views

Naive Bayes / SVM classifiation - min. number of records (Python)

I am doing text classification with Python. I have around 120 records with 2 columns: text class I tokenize, stem and lematize the words, I also did some of my own text preprocessing. When I run the ...
1
vote
0answers
13 views

Populating Knowledge Base - Stanford DeepMind Alternatives

I am dealing with the task to extract structured information from domain-specific unstructured documents. The end goal is to obtain a reliable, queryable system, i.e. in the form of a chat-bot or ...
1
vote
2answers
39 views

Clustering Small Text Descriptions

Im presented with a unique text classification problem. Im given a list of descriptions each containing 3-8 words. I know that there are some descriptions that are nearly the same, but the majority ...
1
vote
2answers
28 views

How can you make use of Json format data?

I want to obtain data from https://petition.parliament.uk/petitions/250967 but the data format is Json. I am new to data mining and I would like to know if there is a way to convert this data into an ...
0
votes
0answers
80 views

Unsupervised learning, Python, text clustering

I want to do unsupervised learning. As I understand with such learning we don't know the clusters before, right? I read about k-means alghoritm, followed mainly these two articles: The first article. ...
1
vote
0answers
21 views

PDF/Text to csv table

I have very little python experience but I have been wanting to get into data science and thought I would start with pdf/text mining. Because it's something I need right now anyway. I have a list of ...
0
votes
0answers
12 views

Looking for recommendations on auto-tagging approaches

Basically looking for approaches to solve the stackoverflow question tagging problem. I have seen a few papers already but in case I have missed something - asking here too. For anyone interested, ...
0
votes
0answers
18 views

how to analyse open ended responses

I have a dataset in which 5 fields are open-ended responses. I need to find interesting insights from this data. My approach is: create a word2vec and then form different clusters and make these ...
1
vote
0answers
23 views

Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?

I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
0
votes
0answers
19 views

Latent Dirichlet Allocation in R, topicmodels using VEM algorithm or Gibbs Sampling mixing tm and topicmodels library or WarpLDA from text2vec?

If I am trying to classify 230k text abstracts, which option would be better and more precise when aplying LDA?
3
votes
2answers
29 views

Classifying dates in sentences

Let's say I have a sentences that goes like this: Hi, how are the kids. I will be going to Los Angeles next Friday and will come back the following Monday. If the date today is October 16 (...
0
votes
1answer
30 views

Information Extraction/Semantic Search for long, unstructured documents

I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store ...
0
votes
0answers
9 views

How to approach the topical text categorization of a small collection of short texts?

I have a set of 200 very short documents, between 1 and 20 words each. One of my colleague would like to classify each of these documents in three predefined topics (let's call them "A", "B", and "C"...
0
votes
0answers
30 views

Collecting structured data from HTML source code: A generalized way

I am working on a task to build a generic function to extract some specific fields from HTML source code. The fields we want are such as product title, price, quantity and shipment The generic ...
0
votes
1answer
47 views

Finding cosine similarity score

I have a dataframe that looks like this: sentence intent hi greeting hello greeting buy this buy whats up conversation . . What I'd like ...
0
votes
1answer
26 views

Why is TF IDF output lognormal?

I ran a TF IDF algorithm and the result of predicted similarities using cosine similarity is a log-normal distribution. Is this a feature of the algorithm (e.g., all logit probabilities are log-...
0
votes
0answers
14 views

POS extraction using CoreNLP

I have a corpus of windows related documents, for which I need to extract nouns and verbs. However, it is required that I keep certain windows specific words such as "inline hooking", "instruction ...
1
vote
1answer
30 views

Can you suggest a better to organise the data to generate frequent itemsets?

I have a data of a bag of words in a document. The data has 3 coulumns : document number, word number, count of the word in the number. I am supposed to generate frequent itemsets of particular size. ...
0
votes
1answer
26 views

appending file text into list with file name in python

Here 20_newsgroups is a folder which contain 20 folder and each folder contain some file , i just want to make datasets containind words and file name its gives Error ...
1
vote
1answer
18 views

Infer family type, size from reviews

I have a bunch of reviews: ...
0
votes
0answers
21 views

Probabilistic model of selecting subsets of words from documents?

Is there an existing probabilistic model that deals with the selection of subsets of words from a corpus of documents? Imagine a stack of documents where a subset of the words in each document has ...
0
votes
0answers
60 views

how to calculate coherence score in topic model

I am trying to calculate coherence score in topic modeling. I am following this Github link So there I need to use the preprocessed wiki and news. I got 3 questions: if the domain that I have ...
0
votes
0answers
14 views

Can we treat sentiment score of the review text as rating of the product?

I have a review text of the different products, And I need the rating of the product. So can we use sentiment score as the rating of the product.
0
votes
0answers
11 views

Is it possible to build an intelligent lead classifier with just a few training units

I want to build a lead classifier for my Master Thesis and wanted to ask for an assessment of feasibility. Here are the key points: (1) We have 15 customers and about 100 opportunities of which we ...
0
votes
2answers
162 views

NLP to detect duplicates for very technical language

I have the following scenario, to detect duplicate products based on the description fields. The Description Field contains product technical name, dimensions, characteristics. My model needs to ...
1
vote
0answers
29 views

Sentiment Analysis for Q&A based reviews

I'm a self-learning ML enthusiast and I recently started learning NLP and performing Sentiment Analysis on imdb, yelp, amazon datasets(using Python). I came across a dataset where the reviews were in ...
1
vote
1answer
27 views

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them?

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them? That is, to get results which say that, for example, the ...
0
votes
1answer
30 views

Which are the appropriate prameters for lda modeling?

I try to implement in R test for appropriate metrics for lda. Here the way I try to use LDA ...
0
votes
0answers
11 views

What options do I have for measuring similarities by using vectors generated from texts?

I have a data set which contains vectors generated from subtitles, I want to measure the similarity between each pair of the observation. Now I have tried L1, L2, cosine similarity and Mahalanobis ...
2
votes
0answers
33 views

Text extraction / mining from specific templates (ML)

believe me if I say that I have read basically all the threads in this website regarding this subject. A lot of them have a similar title, but the problem is somehow different. Small context: just ...
2
votes
1answer
28 views

How to segment old digitized newspapers into articles

I'm working on a large corpus of french daily newspapers from the 19th century that have been digitized and where the data are in the form of raw OCR text files (one text file per day). In terms of ...
0
votes
0answers
224 views

Sentiment Analysis of News Headlines

I'm trying to do sentiment analysis of News Headlines about a particular subject mentioned in it. Initially, I used TextBlob library for sentiment analysis to ...
0
votes
0answers
19 views

Clustering text documents from multiple sources

Let's say I have a set of text documents. Half of the documents are concise social media posts containing a lot of shorthand, and the other half are long news articles. Also, half of the documents ...
6
votes
2answers
81 views

How to compare different similarity measurements in text clustering?

I have a dataset which contains vectors generated from subtitles (each column represents a genre, each row is a movie name), my purpose is to find the most similar movie titles, I want to use ...
2
votes
0answers
31 views

pros and cons of lexical vs machine learning methods for text mining

I wanted to know what are the pros and cons are of using lexical methods and machine learning methods for classifying texts based topic. I have used a simple method of mining documents related to a ...