Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [text-mining]

Refers to a subset of data mining concerned with extracting information from data in the form of text by recognizing patterns. The goal of text mining is often to classify a given document into one of a number of categories in an automatic way, and to improve this performance dynamically, making it an example of machine learning. One example of this type of text mining are spam filters used for email.

Filter by
Sorted by
Tagged with
2 votes
1 answer
21 views

Text Classification with unlimited labels, Text Extraction?

I'm looking to use ML to read in a blob of text, and extract a name from that text blob. (The blob is from an OCR result from an iPhone) The text blob varies in size, but the name is always present in ...
Matthew Knippen's user avatar
0 votes
0 answers
21 views

Insights about W0rd2Vec

As per my knowledge, Word2Vec is belongs to non-contextual embedding technique. this have only semantic relationship between words. We can implement Word2Vec, either in CBoW or skip-gram model. but i ...
Tovlk's user avatar
  • 43
0 votes
0 answers
36 views

Interpreting Perplexity, U_mass coherence and Cv score trends for a Latent Dirichlet Allocation Model

I'm running an LDA model through gensim. To my understanding, closer the u_mass coherence score is to zero, higher is the interpretability of the topics that come up. I'm getting the u_mass coherence ...
Sarthak's user avatar
0 votes
0 answers
17 views

Handwritten text to digital versions - software, libraries, options

What types of (presumably machine learning) software/libraries exist for taking handwritten text in tables into a digital format? The tables may not always be the same. So I assume it might be fairly ...
Socorro's user avatar
  • 111
0 votes
0 answers
11 views

Why does TF-IDF work in TfidfVectorizer?

As I understand TF-IDF, the IDF value of the word "art" = log_e(3/1) + 1 because there are 3 documents in the data set and the word "art" appears once. But after I use the print ...
Khang Khang's user avatar
0 votes
0 answers
8 views

How to pre-processed text to keep dashed words?

I'm working on compiling reviews for movies and analysing it in Orange. I've found that words like "r-rated" get converted to "r" and "rated". I've messed around a bit ...
Rasmus's user avatar
  • 1
0 votes
1 answer
80 views

Clustering words with similar meanings

What methods are there to cluster words/word phrases with similar meanings together from a list of words/word phrases?
ros's user avatar
  • 1
0 votes
1 answer
53 views

What are the different ways of Visualizing Textual Data?

I've recently scraped some textual data from the web and after removing stop-words and punctuations/symbols etc. created a WordCloud to get the most frequent words. ...
Fardeen Khan's user avatar
0 votes
1 answer
137 views

Why custom training a Spacy model runs only the Initializing pipeline but the Training pipeline is not running?

I am training a custom NER model with Spacy version 3.5.0 using some dummy data. My entire code and dummy data is given below. This is exact same code give in the 2nd half of this link. The code is ...
SageMaker's user avatar
  • 185
3 votes
4 answers
2k views

How to automatically classify a sentence or text based on its context?

I have a database of sentences which are about different topics. I want to automatically classify each sentence with the one or more relevant tags based on the context of the sentence as shown below: ...
SageMaker's user avatar
  • 185
0 votes
2 answers
99 views

NLP vs Keyword-Search. which one is the best?

I have constructed a natural language processing (NLP) model with the aim of identifying technology keywords within text. The model is trained on a large dataset that contains over 400,000 phrases and ...
Lakshitha Samod's user avatar
1 vote
1 answer
135 views

Quantitatively evaluate similarity between two corpus of texts

I want to assess how similar, or different two corpora are, and if the similarity is statistically significant. Something close to a Kolmogorov–Smirnov test in statistics, but for text data. For ...
Pujan Paudel's user avatar
0 votes
1 answer
298 views

Generate similar text based on category or the similar texts

I'm trying to generate the similar text based on the category or to generate text by combining similar texts into the new text. I was checking multiple nlp tasks like question generation, but they don'...
Crn's user avatar
  • 1
1 vote
1 answer
69 views

Abstracted text summarisation and generation from weighted keywords

Suppose I have a list of weighted keywords/phrases, such as "solar panel", "rooftop", etc. The weights are in [0,1] with higher weights indicating a stronger preference for ...
Jeff's user avatar
  • 113
2 votes
2 answers
52 views

Finding useful noun 2-grams?

Q: How can I find noun 2-grams in the English language (e.g., "roller coaster", "test tube")? Better yet, how can I find them with proportions? Ultimate goal: Generate distinct ...
lowndrul's user avatar
  • 121
0 votes
0 answers
74 views

Is it possible to use text Auto Encoders without text generation?

I have a use case where I have large texts, and a lot of it. Pretty often the sequence length exceeds 1000 tokens. I need a lower dimensional compression of the texts as an input for a classifier. The ...
thijsvdp's user avatar
  • 101
1 vote
1 answer
137 views

what kind of algorithm should I use to classify the text data example given?

What kind of classification or learning algorithm that suits this kind of data example If I have to build a model using the given key words then predict column B and then to column A? what kind of ...
User123456's user avatar
0 votes
1 answer
27 views

Speech to text to microphone

Is it possible to use an artificial intelligence voice as a microphone in real time? I would like something where I could speak and it would be spoken by an AI, such as Cortana's voice, and pass it to ...
jacoves's user avatar
0 votes
1 answer
29 views

Using relative or absolute frequencies to estimate group differences in texts

My objective is to estimate differences between how five political parties use moral words in their tweets and speeches. To that end, I have a dictionary that I pass to each tweet text / audio ...
Alberto Agudo Dominguez's user avatar
0 votes
0 answers
31 views

Procedure or term for analyzing transcribed text and returning bulleted output

I am attempting to analyze transcribed text from an audio file to group bullet points based on known key phrases in the text. Example: I have verbally stated the following keywords in the text, which ...
Ryan Watts's user avatar
2 votes
2 answers
539 views

Suggestions for guided NLP online courses - Beginner 101

I would like to know from the data science community here for suggestions on nlp courses. I am new to NLP area and would like to take up a course which covers from basic to advanced concepts such as ...
The Great's user avatar
  • 2,565
2 votes
2 answers
2k views

How to find common patterns in thousands of strings?

I don't want to find "abc" in strings ["kkkabczzz", "shdirabckai"] Not like that. But bigger patterns like this: If I have to _________, then I will ___. ["If I have ...
Mohit Gangrade's user avatar
1 vote
1 answer
856 views

Building a graph out of a large text corpus

I'm given a large amount of documents upon which I should perform various kinds of analysis. Since the documents are to be used as a foundation of a final product, I thought about building a graph out ...
kevin_was_here's user avatar
0 votes
1 answer
213 views

How to deal with one output for multiple inputs?

Hei! I want to train a model, that predicts the sentiment of news headlines. I've got multiple unordered news headlines per day, but one sentiment score. What is a convenient solution to overcome the ...
nilosch's user avatar
0 votes
1 answer
295 views

Read corpus from a csv file in Orange3

I have twitter text as an Excel file: every line is one one tweet. How do I view this corpus in Orange3? I don't understand why I can't simply see this corpus. As you can see in the image below, the ...
Huy Truong's user avatar
3 votes
2 answers
378 views

NLP text representation techniques that preserve word order in sentence?

I see people are talking mostly about bag-of-words, td-idf and word embeddings. But these are at word levels. BoW and tf-idf fail to represent word orders, and word embeddings are not meant to ...
Student's user avatar
  • 411
1 vote
1 answer
582 views

How do you handle the free-text fields in tabular data in ML/DL?

While we see a number of cases where the input data is only a single text fields (for the X variable) in NLP tasks, e.g. a tweet with a sentiment label being the only numerical field. But how do you ...
Student's user avatar
  • 411
2 votes
1 answer
46 views

Measuring The Discriminating Power of Clustering

I am measuring the "proximity" of certain words (proper nouns) within a certain text (Silmarillion) by determining the occurrences of these words within the text and, via binning, creating a ...
Matthias's user avatar
  • 121
3 votes
2 answers
574 views

How to do sentence segmentation without loosing sentence's subject?

I have some text with different lengths, I want to split it into separate clauses but I also want to preserve the subject For example; ...
Darkstar Dream's user avatar
0 votes
1 answer
1k views

Should we clean text data before applying Vader for getting sentiment

What I meant by data cleaning is that Removing Punctuations Lower Casing Removing Stop words Removing irrelevant symbols, links and emojis According to my knowledge, things like Punctuations, ...
Yeshan Santhush's user avatar
0 votes
0 answers
110 views

Unsupervised clustering for Text labelling

I have millions semi structured text descriptions of a job requirement. Which needs to be labelled, such as the number of hours, years of experience required, shifts, certifications, licensure etc., ...
Timoth Dev A's user avatar
0 votes
0 answers
91 views

How to get fine-grained sentiment score from text data under unsupervised learning?

In my experience I have only used LSTM models to do sentiment classification tasks on text data under supervised learning. For example, the imdb dataset from keras which is a binary classification ...
user900476's user avatar
0 votes
2 answers
707 views

extracting data from unstructured pdfs

I have about 200,000 PDFs made up of 20 different designs. i.e In an organization, different (20) departments issue monthly award submission requirements. Each department has its own document format. ...
capiono's user avatar
  • 105
2 votes
2 answers
244 views

How to improve language model ex: BERT on unseen text in training?

I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure. However, when I remove a ...
IS92's user avatar
  • 123
1 vote
1 answer
440 views

How to evaluate triple extraction in NLP?

I am current NLP work, I am extracting triples using triple extraction function in Stanford NLP and Spacy libraries. I am looking for a good method to evaluate how good the extraction has been? Any ...
SageMaker's user avatar
  • 185
1 vote
2 answers
38 views

Automatically finding business opportunities in text documents

I am new to machine learning and NLP. I am exploring the possibility of using one of these approaches to automatically examine a large collection of text documents and determine, first of all, if they ...
ProjK's user avatar
  • 11
0 votes
1 answer
96 views

Text clustering model on small dataset

Is there any way to run any clustering model on a small dataset with 290 text records (minimum character size is 100)?
SaNa's user avatar
  • 129
1 vote
0 answers
13 views

I'm looking for some scripts or tools that can take some text, understand and generate multichoice questions, cloze deletion, pop quiz [closed]

I'm a fairly entry level coder in python but I was hoping for some guidance on if, or how I could load a block of text to generate random questions to test comprehension. My goal is to conduct a large ...
Jay Best's user avatar
1 vote
0 answers
114 views

How to extract numerical information from text descriptions

I have an attribute that is the description of an operation (i.e description of a building consent), I need to translate this to a mathematical operation. I need to find out the new number of dwelling ...
Mohsen Sichani's user avatar
0 votes
0 answers
165 views

NLP conversation data - Pre-processing steps

I have text data, so data that has been transcribed from conversations from employees to customers. So each call has a recording that has been transcribed to text. I am looking to do some analytics ...
theassassin11's user avatar
1 vote
0 answers
48 views

Unstructured data not template based to structured data

I am working on a project where my goal is to extract data from a large diversity of pdf that does not follow a template (i.e unstructured data not template based). My ORC part works well and now I am ...
Ketchup's user avatar
  • 29
0 votes
0 answers
40 views

Problem Direction - Text Data - Conversation Classification

The problem I have a problem where I have text data that has been transcribed from a conversation. These conversations have been marked as a pass or fail in terms of compliance by a person, ie they ...
theassassin11's user avatar
1 vote
1 answer
155 views

Methods for finding characteristic words for a group of documents in comparison to another group of documents?

I'm working on a problem of anomaly detection, where at the end of the anomaly detection I will have a group of documents consisting of a title of each object that was flagged as anomalous. At the ...
kspr's user avatar
  • 133
2 votes
2 answers
113 views

How to stop a text-classification model from depending on only couple of the words from input text instead of entire sentence?

I have a text classification deep-learning model, which takes in a text and outputs a softmax probability. I am using glove embeddings to represent my input text in numerical form for the DL model. ...
Naveen Reddy Marthala's user avatar
2 votes
1 answer
380 views

Clustering Strings Based on Similar Word Sequences

In my dataset I have a feature having below data : Input Feature Brain Dementia Routine(Comfortone) Morning Check Dementia Brain-Routine(Comfortone) Brain MRA Routine (Comfortone) Brain-Dementia/...
user125887's user avatar
0 votes
2 answers
1k views

calibrated classifier ValueError: could not convert string to float

Dataframe: ...
SaNa's user avatar
  • 129
0 votes
1 answer
231 views

how to filter out and discard irrelevant tweets in simplest way possible

I have lot of tweets and from which i need to filter out and discard irrelevant tweets. the criteria for a tweet to be irrelevant is very simple. if all that a tweet has is emojis or a single hastag ...
Naveen Reddy Marthala's user avatar
0 votes
2 answers
310 views

How to stay up to date in NLP and use the best approaches?

There are many fast advancements in NLP field, BERT, RoBERTa, ALBERT, and XLNe, and no one can check the news or papers daily. Is there any way or site that keeps track of all these new developments ...
Thomas Lee's user avatar
1 vote
1 answer
133 views

Choice of the number of topics (clusters) in textual data

I have a social science background and I'm doing a text mining project. I'm looking for advice about the choice of the number of topics/clusters when analyzing textual data. In particular, I'm ...
Himan's user avatar
  • 11
0 votes
2 answers
52 views

Text mining match in Python [closed]

I have one column called A and one column called B where column B is a more detailed ...
Sweety Vaidya's user avatar

1
2 3 4 5
12