Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
1
vote
0answers
13 views

Could I use the harmonic mean method to determine k number of topics when applying Latent Dirichlet Allocation using text2vec?

I am using text2vec to apply LDA on 230k docs reduced to 800 terms aprox. Is it okay to use the harmonic mean to approximate the marginal likelihood in order to mention the best topic number when that ...
0
votes
0answers
16 views

Latent Dirichlet Allocation in R, topicmodels using VEM algorithm or Gibbs Sampling mixing tm and topicmodels library or WarpLDA from text2vec?

If I am trying to classify 230k text abstracts, which option would be better and more precise when aplying LDA?
2
votes
2answers
23 views

Classifying dates in sentences

Let's say I have a sentences that goes like this: Hi, how are the kids. I will be going to Los Angeles next Friday and will come back the following Monday. If the date today is October 16 (...
0
votes
1answer
20 views

Information Extraction/Semantic Search for long, unstructured documents

I am stuck with a particular task of information extraction. I have a few hundred, long (5-35 pages) pdf, doc and docx project documents from which I seek to extract specific information and store ...
0
votes
0answers
9 views

How to approach the topical text categorization of a small collection of short texts?

I have a set of 200 very short documents, between 1 and 20 words each. One of my colleague would like to classify each of these documents in three predefined topics (let's call them "A", "B", and "C"...
0
votes
0answers
27 views

Collecting structured data from HTML source code: A generalized way

I am working on a task to build a generic function to extract some specific fields from HTML source code. The fields we want are such as product title, price, quantity and shipment The generic ...
-1
votes
0answers
8 views

Matching phrase using TF-IDF and cosine similarity

I have a dataframe that looks like this: ...
0
votes
1answer
22 views

Finding cosine similarity score

I have a dataframe that looks like this: sentence intent hi greeting hello greeting buy this buy whats up conversation . . What I'd like ...
0
votes
1answer
23 views

Why is TF IDF output lognormal?

I ran a TF IDF algorithm and the result of predicted similarities using cosine similarity is a log-normal distribution. Is this a feature of the algorithm (e.g., all logit probabilities are log-...
0
votes
0answers
10 views

POS extraction using CoreNLP

I have a corpus of windows related documents, for which I need to extract nouns and verbs. However, it is required that I keep certain windows specific words such as "inline hooking", "instruction ...
1
vote
1answer
17 views

Can you suggest a better to organise the data to generate frequent itemsets?

I have a data of a bag of words in a document. The data has 3 coulumns : document number, word number, count of the word in the number. I am supposed to generate frequent itemsets of particular size. ...
0
votes
1answer
15 views

appending file text into list with file name in python

Here 20_newsgroups is a folder which contain 20 folder and each folder contain some file , i just want to make datasets containind words and file name its gives Error ...
0
votes
0answers
8 views

Orange Error During Installing Text Add On [migrated]

I am trying to install the Text add-on (version 0.7.3) for Orange (version 3.23) on my Win10, but I am getting the following error during the building "ufal_udpipe": ...
1
vote
1answer
17 views

Infer family type, size from reviews

I have a bunch of reviews: ...
0
votes
0answers
20 views

Probabilistic model of selecting subsets of words from documents?

Is there an existing probabilistic model that deals with the selection of subsets of words from a corpus of documents? Imagine a stack of documents where a subset of the words in each document has ...
0
votes
0answers
41 views

how to calculate coherence score in topic model

I am trying to calculate coherence score in topic modeling. I am following this Github link So there I need to use the preprocessed wiki and news. I got 3 questions: if the domain that I have ...
0
votes
0answers
13 views

Can we treat sentiment score of the review text as rating of the product?

I have a review text of the different products, And I need the rating of the product. So can we use sentiment score as the rating of the product.
0
votes
0answers
11 views

Is it possible to build an intelligent lead classifier with just a few training units

I want to build a lead classifier for my Master Thesis and wanted to ask for an assessment of feasibility. Here are the key points: (1) We have 15 customers and about 100 opportunities of which we ...
0
votes
2answers
56 views

NLP to detect duplicates for very technical language

I have the following scenario, to detect duplicate products based on the description fields. The Description Field contains product technical name, dimensions, characteristics. My model needs to ...
1
vote
0answers
23 views

Sentiment Analysis for Q&A based reviews

I'm a self-learning ML enthusiast and I recently started learning NLP and performing Sentiment Analysis on imdb, yelp, amazon datasets(using Python). I came across a dataset where the reviews were in ...
1
vote
1answer
24 views

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them?

Given two large corpora of text from different sources, is there an accepted way to get differences in vocabulary (n-grams) between them? That is, to get results which say that, for example, the ...
0
votes
1answer
24 views

Which are the appropriate prameters for lda modeling?

I try to implement in R test for appropriate metrics for lda. Here the way I try to use LDA ...
0
votes
0answers
11 views

What options do I have for measuring similarities by using vectors generated from texts?

I have a data set which contains vectors generated from subtitles, I want to measure the similarity between each pair of the observation. Now I have tried L1, L2, cosine similarity and Mahalanobis ...
2
votes
0answers
31 views

Text extraction / mining from specific templates (ML)

believe me if I say that I have read basically all the threads in this website regarding this subject. A lot of them have a similar title, but the problem is somehow different. Small context: just ...
2
votes
1answer
19 views

How to segment old digitized newspapers into articles

I'm working on a large corpus of french daily newspapers from the 19th century that have been digitized and where the data are in the form of raw OCR text files (one text file per day). In terms of ...
0
votes
0answers
87 views

Sentiment Analysis of News Headlines

I'm trying to do sentiment analysis of News Headlines about a particular subject mentioned in it. Initially, I used TextBlob library for sentiment analysis to ...
0
votes
0answers
16 views

Clustering text documents from multiple sources

Let's say I have a set of text documents. Half of the documents are concise social media posts containing a lot of shorthand, and the other half are long news articles. Also, half of the documents ...
6
votes
2answers
67 views

How to compare different similarity measurements in text clustering?

I have a dataset which contains vectors generated from subtitles (each column represents a genre, each row is a movie name), my purpose is to find the most similar movie titles, I want to use ...
2
votes
0answers
16 views

pros and cons of lexical vs machine learning methods for text mining

I wanted to know what are the pros and cons are of using lexical methods and machine learning methods for classifying texts based topic. I have used a simple method of mining documents related to a ...
1
vote
1answer
48 views

Data Entry Automation with ML

I am working on a data entry task with approximately 6000 entries to go over. The source comes in the form of a string and can look something like this: Air Canada B737 FFS From this I can ...
1
vote
0answers
24 views

Grouping paragraphs of text by type

I'm trying to parse some text, and extract data from it. Typical NLP problem. However the text contains different sections, and I know that the keywords of interest are in specific sections, but all ...
1
vote
0answers
26 views

How to classify product by specific category without machine learning?

I am working on a product classification problem which I have to identify product category. Say for the category, there are 5 levels (Big / medium / small / detail / double detail) 5 million ...
0
votes
0answers
11 views

Clustering of words based on ability to predict variation in other variables

I have a data set of about 1000 observations like so: ...
0
votes
0answers
19 views

Class Size Imbalance for LDA or any other Content based analysis

I am running some content analysis studies on my dataset which has two different classes, and each class has a respective list of the document I am analyzing. I compare the LDA topic model inference ...
1
vote
1answer
25 views

Technical term for using regular expressions to classify text?

Background I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext. He wants to classify the claims as either having been "approved", "denied", ...
0
votes
0answers
9 views

Classifying Short Texts with Spatial Features

I have a dataset of short texts (like tweets) in addition there's some geographical data attached to each tweet - coordinates, whether it was made on the road, street, outside or in the building, ...
1
vote
1answer
121 views

DBSCAN clustering on document [updated]?

I am new in topic modeling and text clustering domain and I am trying to learn more. I would like to use the DBSCAN to cluster the text data. There are many posts and sources on how to implement the ...
1
vote
1answer
35 views

Build text complexity model based on complex examples

I try to build the user specific model which predicts whether arbitrary English text is complex for particular user or not. Having the complex and easy text samples allows to build such model but what ...
0
votes
0answers
13 views

How to handle variables with text and number?

This is a post with two related questions in one. The first question is: What is the correct procedure when I have variables with different kind of information? Imagine you have a column which has ...
1
vote
1answer
61 views

Find specific topics with topic modelling

I am looking for a way to classifiy text automatically by specific topics, i don´t have labeled data. Is this a possible/usual method of achieving this? If not, what would be better? Topic Modelling ...
1
vote
1answer
38 views

Emotional tension score in sentences

I am beginner in natural language processing and my goal is to find a way to score sentences based on their emotional tension. More specifically, I would like to know to what degree a sentence ...
0
votes
0answers
13 views

How to interpret the transition and feature weights in a Conditional Random Field model?

Conditional Random Fields model have been a popular method for Named Entity Recognition as it accounts for statistical dependencies between entities and can include observed features that can aid with ...
0
votes
2answers
43 views

Targeted information extraction / focused extractive summarization

I have a large collection of project manuals, each with a large number of pages. Each manual contains some form of summary paragraphs, although these are not necessarily similar in structure or format ...
0
votes
1answer
122 views

Which libraries in Python are there in NLP to tokenize the Hindi sentence?

For English language there are libraries like NLTK, CoreNLP which are used for Text Normalization, Word Tokenization and Detokenization, Sentence Splitting etc. Like English, is there any library to ...
2
votes
0answers
35 views

Assigning tags to posts using predefined set of tags

I want to tag the text of a post with a predefined set of tags. A post could have multiple tags such as health, addiction, etc. I want to recommend up to $5$ tags. Total of $60$ tags is present. ...
1
vote
1answer
31 views

What's the fastest way to do a text analysis over user reviews on a website for a beginner? [closed]

I want to analyse user reviews for certain products as part of a research project without having to learn analytics from scratch, as my requirement is temporary. I need to do the following: The user ...
0
votes
1answer
31 views

What is the best approach to perform information extraction from tourist reviews using NLP, DL?

I am interested in performing some information extraction from tourist reviews about different places. I have data of 50 different places and around 300-400 reviews about each of them and I would ...
1
vote
1answer
87 views

Does it make sense to use TF-IDF to extract most important tokens from a corpus?

I have a collection of documents and I'd like to extract the most important words and phrases from the entire corpus. My understanding of TF-IDF is that it is calculated per token per document, so ...
-1
votes
1answer
17 views

Identifying specific words in text

Let's say I have the following text" Is that another kitten playing in the shoes in the top right? I would like my code to extract kitten from that text. Is ...
2
votes
0answers
36 views

Text Mining with Pubmed Widget Orange

When I was running the text mining I did not have an issue for 57 different searches. I was able to retrieve all of the records regardless of how many there were. Until these 2 errors popped up. I ...