Questions tagged [text-mining]
Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.
574
questions
0
votes
0
answers
18
views
Detecting text in binary blobs using ML
Context - I'm an experienced SWE with little expereince in ML, I would like to perform the following:
Given a binary blob (bytearray) of size 256, consisting of random bytes with meaningful text ...
- 101
1
vote
0
answers
14
views
Geolocation Clustering - What is the right approach?
I am working on a research project on illegal dumping within a city & devicing advanced analytics solutions for the waste management company like resource optimisation, trend identification etc ...
2
votes
1
answer
26
views
Text Classification with unlimited labels, Text Extraction?
I'm looking to use ML to read in a blob of text, and extract a name from that text blob. (The blob is from an OCR result from an iPhone)
The text blob varies in size, but the name is always present in ...
- 121
0
votes
0
answers
22
views
Insights about W0rd2Vec
As per my knowledge, Word2Vec is belongs to non-contextual embedding technique. this have only semantic relationship between words.
We can implement Word2Vec, either in CBoW or skip-gram model. but i ...
- 43
0
votes
0
answers
50
views
Interpreting Perplexity, U_mass coherence and Cv score trends for a Latent Dirichlet Allocation Model
I'm running an LDA model through gensim. To my understanding, closer the u_mass coherence score is to zero, higher is the interpretability of the topics that come up. I'm getting the u_mass coherence ...
- 1
0
votes
0
answers
18
views
Handwritten text to digital versions - software, libraries, options
What types of (presumably machine learning) software/libraries exist for taking handwritten text in tables into a digital format? The tables may not always be the same. So I assume it might be fairly ...
- 121
0
votes
0
answers
11
views
Why does TF-IDF work in TfidfVectorizer?
As I understand TF-IDF, the IDF value of the word "art" = log_e(3/1) + 1 because there are 3 documents in the data set and the word "art" appears once. But after I use the print ...
0
votes
0
answers
10
views
How to pre-processed text to keep dashed words?
I'm working on compiling reviews for movies and analysing it in Orange. I've found that words like "r-rated" get converted to "r" and "rated". I've messed around a bit ...
- 1
0
votes
1
answer
209
views
Clustering words with similar meanings
What methods are there to cluster words/word phrases with similar meanings together from a list of words/word phrases?
- 1
0
votes
1
answer
75
views
What are the different ways of Visualizing Textual Data?
I've recently scraped some textual data from the web and after removing stop-words and punctuations/symbols etc. created a WordCloud to get the most frequent words. ...
- 27
0
votes
1
answer
150
views
Why custom training a Spacy model runs only the Initializing pipeline but the Training pipeline is not running?
I am training a custom NER model with Spacy version 3.5.0 using some dummy data. My entire code and dummy data is given below. This is exact same code give in the 2nd half of this link. The code is ...
- 185
3
votes
4
answers
2k
views
How to automatically classify a sentence or text based on its context?
I have a database of sentences which are about different topics. I want to automatically classify each sentence with the one or more relevant tags based on the context of the sentence as shown below:
...
- 185
0
votes
2
answers
117
views
NLP vs Keyword-Search. which one is the best?
I have constructed a natural language processing (NLP) model with the aim of identifying technology keywords within text. The model is trained on a large dataset that contains over 400,000 phrases and ...
- 103
1
vote
1
answer
216
views
Quantitatively evaluate similarity between two corpus of texts
I want to assess how similar, or different two corpora are, and if the similarity is statistically significant. Something close to a Kolmogorov–Smirnov test in statistics, but for text data.
For ...
- 53
0
votes
1
answer
340
views
Generate similar text based on category or the similar texts
I'm trying to generate the similar text based on the category or to generate text by combining similar texts into the new text. I was checking multiple nlp tasks like question generation, but they don'...
- 1
1
vote
1
answer
91
views
Abstracted text summarisation and generation from weighted keywords
Suppose I have a list of weighted keywords/phrases, such as "solar panel", "rooftop", etc. The weights are in [0,1] with higher weights indicating a stronger preference for ...
- 113
2
votes
2
answers
54
views
Finding useful noun 2-grams?
Q: How can I find noun 2-grams in the English language (e.g., "roller coaster", "test tube")? Better yet, how can I find them with proportions?
Ultimate goal: Generate distinct ...
- 121
0
votes
0
answers
94
views
Is it possible to use text Auto Encoders without text generation?
I have a use case where I have large texts, and a lot of it. Pretty often the sequence length exceeds 1000 tokens. I need a lower dimensional compression of the texts as an input for a classifier. The ...
- 101
1
vote
1
answer
157
views
what kind of algorithm should I use to classify the text data example given?
What kind of classification or learning algorithm that suits this kind of data example
If I have to build a model using the given key words then predict column B and then to column A?
what kind of ...
- 21
0
votes
1
answer
28
views
Speech to text to microphone
Is it possible to use an artificial intelligence voice as a microphone in real time? I would like something where I could speak and it would be spoken by an AI, such as Cortana's voice, and pass it to ...
- 1
0
votes
1
answer
35
views
Using relative or absolute frequencies to estimate group differences in texts
My objective is to estimate differences between how five political parties use moral words in their tweets and speeches. To that end, I have a dictionary that I pass to each tweet text / audio ...
0
votes
0
answers
31
views
Procedure or term for analyzing transcribed text and returning bulleted output
I am attempting to analyze transcribed text from an audio file to group bullet points based on known key phrases in the text.
Example: I have verbally stated the following keywords in the text, which ...
- 101
2
votes
2
answers
542
views
Suggestions for guided NLP online courses - Beginner 101
I would like to know from the data science community here for suggestions on nlp courses.
I am new to NLP area and would like to take up a course which covers from basic to advanced concepts such as ...
- 2,607
3
votes
2
answers
2k
views
How to find common patterns in thousands of strings?
I don't want to find "abc" in strings ["kkkabczzz", "shdirabckai"]
Not like that.
But bigger patterns like this:
If I have to _________, then I will ___.
["If I have ...
- 131
1
vote
1
answer
939
views
Building a graph out of a large text corpus
I'm given a large amount of documents upon which I should perform various kinds of analysis. Since the documents are to be used as a foundation of a final product, I thought about building a graph out ...
0
votes
1
answer
235
views
How to deal with one output for multiple inputs?
Hei!
I want to train a model, that predicts the sentiment of news headlines. I've got multiple unordered news headlines per day, but one sentiment score.
What is a convenient solution to overcome the ...
- 1
0
votes
1
answer
339
views
Read corpus from a csv file in Orange3
I have twitter text as an Excel file: every line is one one tweet. How do I view this corpus in Orange3? I don't understand why I can't simply see this corpus. As you can see in the image below, the ...
- 115
3
votes
2
answers
429
views
NLP text representation techniques that preserve word order in sentence?
I see people are talking mostly about bag-of-words, td-idf and word embeddings. But these are at word levels. BoW and tf-idf fail to represent word orders, and word embeddings are not meant to ...
- 411
1
vote
1
answer
638
views
How do you handle the free-text fields in tabular data in ML/DL?
While we see a number of cases where the input data is only a single text fields (for the X variable) in NLP tasks, e.g. a tweet with a sentiment label being the only numerical field.
But how do you ...
- 411
2
votes
1
answer
49
views
Measuring The Discriminating Power of Clustering
I am measuring the "proximity" of certain words (proper nouns) within a certain text (Silmarillion) by determining the occurrences of these words within the text and, via binning, creating a ...
- 121
3
votes
2
answers
653
views
How to do sentence segmentation without loosing sentence's subject?
I have some text with different lengths, I want to split it into separate clauses but I also want to preserve the subject
For example;
...
- 163
0
votes
1
answer
2k
views
Should we clean text data before applying Vader for getting sentiment
What I meant by data cleaning is that
Removing Punctuations
Lower Casing
Removing Stop words
Removing irrelevant symbols, links and emojis
According to my knowledge, things like Punctuations, ...
0
votes
0
answers
92
views
How to get fine-grained sentiment score from text data under unsupervised learning?
In my experience I have only used LSTM models to do sentiment classification tasks on text data under supervised learning. For example, the imdb dataset from keras which is a binary classification ...
- 165
0
votes
2
answers
793
views
extracting data from unstructured pdfs
I have about 200,000 PDFs made up of 20 different designs. i.e In an organization, different (20) departments issue monthly award submission requirements. Each department has its own document format. ...
- 105
2
votes
2
answers
300
views
How to improve language model ex: BERT on unseen text in training?
I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure.
However, when I remove a ...
- 123
1
vote
1
answer
463
views
How to evaluate triple extraction in NLP?
I am current NLP work, I am extracting triples using triple extraction function in Stanford NLP and Spacy libraries. I am looking for a good method to evaluate how good the extraction has been? Any ...
- 185
1
vote
2
answers
38
views
Automatically finding business opportunities in text documents
I am new to machine learning and NLP.
I am exploring the possibility of using one of these approaches to automatically examine a large collection of text documents and determine, first of all, if they ...
- 11
0
votes
1
answer
105
views
Text clustering model on small dataset
Is there any way to run any clustering model on a small dataset with 290 text records (minimum character size is 100)?
- 129
1
vote
0
answers
13
views
I'm looking for some scripts or tools that can take some text, understand and generate multichoice questions, cloze deletion, pop quiz [closed]
I'm a fairly entry level coder in python but I was hoping for some guidance on if, or how I could load a block of text to generate random questions to test comprehension.
My goal is to conduct a large ...
- 11
1
vote
0
answers
124
views
How to extract numerical information from text descriptions
I have an attribute that is the description of an operation (i.e description of a building consent), I need to translate this to a mathematical operation. I need to find out the new number of dwelling ...
- 143
0
votes
0
answers
176
views
NLP conversation data - Pre-processing steps
I have text data, so data that has been transcribed from conversations from employees to customers. So each call has a recording that has been transcribed to text.
I am looking to do some analytics ...
1
vote
0
answers
52
views
Unstructured data not template based to structured data
I am working on a project where my goal is to extract data from a large diversity of pdf that does not follow a template (i.e unstructured data not template based). My ORC part works well and now I am ...
- 129
0
votes
0
answers
43
views
Problem Direction - Text Data - Conversation Classification
The problem
I have a problem where I have text data that has been transcribed from a conversation. These conversations have been marked as a pass or fail in terms of compliance by a person, ie they ...
1
vote
1
answer
172
views
Methods for finding characteristic words for a group of documents in comparison to another group of documents?
I'm working on a problem of anomaly detection, where at the end of the anomaly detection I will have a group of documents consisting of a title of each object that was flagged as anomalous.
At the ...
- 133
2
votes
2
answers
132
views
How to stop a text-classification model from depending on only couple of the words from input text instead of entire sentence?
I have a text classification deep-learning model, which takes in a text and outputs a softmax probability. I am using glove embeddings to represent my input text in numerical form for the DL model.
...
2
votes
1
answer
404
views
Clustering Strings Based on Similar Word Sequences
In my dataset I have a feature having below data :
Input
Feature
Brain Dementia Routine(Comfortone)
Morning Check
Dementia Brain-Routine(Comfortone)
Brain MRA Routine (Comfortone)
Brain-Dementia/...
- 21
0
votes
2
answers
1k
views
0
votes
1
answer
253
views
how to filter out and discard irrelevant tweets in simplest way possible
I have lot of tweets and from which i need to filter out and discard irrelevant tweets. the criteria for a tweet to be irrelevant is very simple. if all that a tweet has is emojis or a single hastag ...
0
votes
2
answers
316
views
How to stay up to date in NLP and use the best approaches?
There are many fast advancements in NLP field, BERT, RoBERTa, ALBERT, and XLNe, and no one can check the news or papers daily. Is there any way or site that keeps track of all these new developments ...
- 155
1
vote
1
answer
170
views
Choice of the number of topics (clusters) in textual data
I have a social science background and I'm doing a text mining project.
I'm looking for advice about the choice of the number of topics/clusters when analyzing textual data. In particular, I'm ...
- 11