Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [text]

Text is a type of data often used in data science projects involving natural language processing.

Filter by
Sorted by
Tagged with
2 votes
1 answer
21 views

Text Classification with unlimited labels, Text Extraction?

I'm looking to use ML to read in a blob of text, and extract a name from that text blob. (The blob is from an OCR result from an iPhone) The text blob varies in size, but the name is always present in ...
Matthew Knippen's user avatar
0 votes
0 answers
17 views

Handwritten text to digital versions - software, libraries, options

What types of (presumably machine learning) software/libraries exist for taking handwritten text in tables into a digital format? The tables may not always be the same. So I assume it might be fairly ...
Socorro's user avatar
  • 111
2 votes
2 answers
248 views

Why is dictionary-based approach a heuristic method?

In How can the accuracy of the dictionary-based approach be measured and improved?, one user says that: dictionary-based approach is a heuristic method Isn't that this approach is a type of rule-...
Ooker's user avatar
  • 123
0 votes
0 answers
22 views

Regarding TextVectorization reserved tokens

From the documentation of TextVectorization: max_tokens: Maximum size of the vocabulary for this layer. This should only be specified when adapting a vocabulary or when setting pad_to_max_tokens=...
Enk9456's user avatar
  • 105
0 votes
0 answers
16 views

Do any text tagging NLP tools use a relational database or anything other than text files to annotate texts?

I am learning about how production systems implement text tagging. MILA has this XML format, and according to ChatGPT, Brat has a text file format, though haven't found much examples of that other ...
Lance's user avatar
  • 273
0 votes
0 answers
28 views

Why did the Double Metaphone algorithm choose to substitute and merge consonants?

Is there any literature describing the decisions behind why the mappings from input text characters to Metaphone hash consonants were made? Why did they choose to leave out vowels? Why did they merge ...
Lance's user avatar
  • 273
0 votes
0 answers
14 views

Highly unbalnced text data giving very low matrics

I have an unbalanced multi-class banking text data with around 76 classes. Classes are badly distributed such as one class which is combination of 240 other different categories, represents 50% of ...
Remrem's user avatar
  • 1
0 votes
0 answers
81 views

How to input or estimate missing text data?

My task requires 32 different columns with 25 beeing independent text data. Deleting nan values, or cutting columns which has less then 20% (of non NaNs) results in reducing dataset to less then 10% ...
Paweł B's user avatar
1 vote
0 answers
19 views

Clustering Similar Articles Using Mixed Data: Seeking Advice and Validation

Question: I'm working on a project where I need to cluster a dataset of articles based on various features, including text, numeric values, and categorical data. I've implemented a clustering approach ...
sara sara's user avatar
0 votes
0 answers
55 views

What dimensional reduction and similarity score work for sentence embeddings created using sentence transformers

I am clustering sentence embeddings for log files, and find anomalies. So, when I create sentence embeddings for logs using sentence transformers. It will create vector of fixed length, which somehow ...
Glinty's user avatar
  • 33
0 votes
0 answers
103 views

Best practice for fine tuning LLM

I have a dataset that i have collect for specific topic the dataset is in these format : raw text (similar to shake spare dataset) where it has no label or input, just text Question and answer ...
Mustafa Alahmid's user avatar
0 votes
1 answer
22 views

Trouble Loading Lines from Text File with Various Encodings

I have been facing difficulties while loading specific lines from a text file. The lines contain characters such as ٹام بیمار ÛÛ’Û” ٹام بیمار ÛÛ’. I have tried using different ...
Abdul Basit Niazi's user avatar
0 votes
1 answer
213 views

Text segmentation problem

I am new to ML and trying to solve problem of text segmentation. I have a transcript of news show and I want to split this transcript into parts by topic. I tried to google and asked chatgpt and found ...
Oleg Bovykin's user avatar
1 vote
1 answer
127 views

How do people usually handle creating an embedding vector of longer texts (32000 characters?

I have a set of podcast episode transcriptions in Arabic. I wish to convert these to embedding vectors so I can run a similarity comparison of them. Here's the summary statistics on the episodes: ...
Stan Shunpike's user avatar
2 votes
1 answer
174 views

Were any LLMs trained on Google books?

An important limiting factor on the performance of large language models, is the amount of training text available. Of course, using e.g. the Gutenberg archive of public domain books is an obvious ...
rwallace's user avatar
  • 149
0 votes
0 answers
20 views

Can this task for phrases be called lemmatization?

I want to 'lemmatize' phrases to dictionary entries. For instance, the following collocates can be standardized to the idiom in the aforementioned link ...
Lerner Zhang's user avatar
0 votes
0 answers
96 views

Creating variations of prompts for ChatGPT

I am developing a fine tune model to emulate a tech support chatbot based on my given information. I am struggling to create a large dataset (aiming for 1000 prompt/completion pairs), does anyone have ...
user624's user avatar
1 vote
1 answer
67 views

Dictionary-based text analysis- dealing with length

I am working on an analysis using a dictionary-based text-as-data approach. I have a dataset of texts (n=1200), and I am applying a dictionary of 50 words (I tokenize the text with each word being one ...
Iamembarassed123's user avatar
1 vote
1 answer
103 views

how to extract common aspects from text using deep learning?

Can you suggest me some papers to read about deep learning models that find patterns/similarities between different texts? What I have is a set of reviews with the following categories for each review:...
Alberto De Benedittis's user avatar
1 vote
1 answer
14 views

Predictive value of short text fields

I am working on a classification model using one of the following three algorithms: RandomForestClassifier, a TensorFlow model and a LogisticRegression model. The data set I am working with has a ...
str31's user avatar
  • 13
0 votes
0 answers
17 views

Text analysis - "interpret" bag of words

I wanted to try to create a machine learning tarot card reader. Setting up a simulated draw of tarot cards and their associated meanings is straight-forward enough. However, the listed applications I ...
Arash Howaida's user avatar
0 votes
1 answer
29 views

Using relative or absolute frequencies to estimate group differences in texts

My objective is to estimate differences between how five political parties use moral words in their tweets and speeches. To that end, I have a dictionary that I pass to each tweet text / audio ...
Alberto Agudo Dominguez's user avatar
1 vote
1 answer
33 views

Next-word Generation in Tabular Dataset

I'll build next-word generation using Tensorflow to predict address mapping. But, I saw many tutorial, next-word generation use long-text narration for its training dataset. But, I have dataset ...
Mico S's user avatar
  • 41
2 votes
2 answers
539 views

Suggestions for guided NLP online courses - Beginner 101

I would like to know from the data science community here for suggestions on nlp courses. I am new to NLP area and would like to take up a course which covers from basic to advanced concepts such as ...
The Great's user avatar
  • 2,565
1 vote
1 answer
636 views

How to calculate Pointwise Mutual Information (PMI) when working with multiple ngrams

Pointwise Mutual Information or PMI for short is given as Which is the same as: Where BigramOccurrences is number of times bigram appears as feature, 1stWordOccurrences is number of times 1st word ...
SequentialResidential's user avatar
1 vote
1 answer
118 views

What is the logic/algorithm behind 'did you mean' suggestion by search engines, command suggestion in command prompt like git?

For eg. https://stackoverflow.com/questions/307291/how-does-the-google-did-you-mean-algorithm-work this is the logic behind google's did you mean algorithm - used for spell correction suggestion. What ...
jarvis's user avatar
  • 11
3 votes
2 answers
378 views

NLP text representation techniques that preserve word order in sentence?

I see people are talking mostly about bag-of-words, td-idf and word embeddings. But these are at word levels. BoW and tf-idf fail to represent word orders, and word embeddings are not meant to ...
Student's user avatar
  • 411
1 vote
1 answer
456 views

How to down\up sample text?

I have data set of 5566 samples - one column is the text of the recipe description and the other is what tax class is it. I wish to make a classifier that would classify receipts using ML only. I have ...
JamseGoldman's user avatar
1 vote
1 answer
214 views

Find best features for text classification with Countvectorizer Python

I'm facing a text classification problem where the algo is human-made but impacted by keywords. Hence I can't use any ML model but I certainly can take a data-science driven approach to find the best ...
Waroulolz's user avatar
  • 113
1 vote
1 answer
28 views

Partially labelled open-class classification problem with heavy overlap

Let's say we have a corpus of text, including discussions about movies and about sports. Unsupervised clustering would typically cluster into the two topics of discussion. However, we are interested ...
basiliskcompliantentity's user avatar
1 vote
1 answer
306 views

Matching 2 keywords list using NLP

I have two lists and I want to identify which elements are common (same or similar in meaning or context) in the list. Which NLP algorithm we should use. ...
Mohit Tripathi's user avatar
0 votes
2 answers
4k views

How to set vocabulary size, padding length and embedding dimension in LSTM network?

Usually in a LSTM network, we have certain parameters that need to be set before the model can begin training. I am specifically talking about vocabulary size, padding length and embedding dimension. ...
spectre's user avatar
  • 2,105
0 votes
2 answers
292 views

Keywords extraction for business rule text classification

I would like to classify texts without using any ML model. My idea is to find a list of keywords that I would assign to each class. Then when I need to classify a new text, I can compare it with my ...
Waroulolz's user avatar
  • 113
1 vote
1 answer
155 views

Why not rule-based semantic role labelling?

I have recently found some interest in automatic semantic role labelling. Most introductory texts (e.g. Jurafsky and Martin, 2008) present approaches based on supervised machine learning, often using ...
thesofakillers's user avatar
0 votes
0 answers
307 views

Text classification length

I have a set of text examples I need to learn as class A, and they are of varying lengths, say 10 sentences to 1 sentence long. I have to parse a document to find those strings of text that match one ...
superqd's user avatar
  • 115
0 votes
0 answers
165 views

NLP conversation data - Pre-processing steps

I have text data, so data that has been transcribed from conversations from employees to customers. So each call has a recording that has been transcribed to text. I am looking to do some analytics ...
theassassin11's user avatar
5 votes
1 answer
2k views

How to evaluate the similarity of two columns containing strings?

I am new to text processing and stuck on a problem to identify the similarity of columns. To detail the problem, consider we have two columns with string values: ...
Rachit Tayal's user avatar
0 votes
0 answers
40 views

Problem Direction - Text Data - Conversation Classification

The problem I have a problem where I have text data that has been transcribed from a conversation. These conversations have been marked as a pass or fail in terms of compliance by a person, ie they ...
theassassin11's user avatar
0 votes
0 answers
31 views

How do I work with noisy real world text data for text classification?

I have a topic classification model built upon Bert, when I deploy my model people input strings of a random nature like : "aaaaaa" "aaa bbb" "ab ab ab" and so on. My ...
tester57 qwerr's user avatar
0 votes
1 answer
42 views

Working with three types of data: numeric (integer, floats), images, and text for prediction

So I have three types of data (in title) and am wondering how I can combine the data. The target is numeric (price). My idea is to perform feature extraction on both the images and text, which would ...
new_account_49's user avatar
1 vote
2 answers
140 views

Topic modelling on long documents: intra document clustering first

I have a collection (around 1000) of very noisy, similar documents, that are each very long (>10 pages - 600 paragraphs) with multiple subsections - I want to perform topic modelling across the ...
James Stirling's user avatar
2 votes
1 answer
49 views

Classifying short strings of text with additional context

I have a list of short strings each identifying a city. Misspellings are very common. The example below shows some of these short strings, along with the correct city they're supposed to match. ...
Jivan's user avatar
  • 165
2 votes
1 answer
3k views

How to impute missing text data?

Lets say I have a dataframe consisting of two text columns. By text, I mean the values in those columns are either sentences/paragraphs. In such a case, how do I handle missing 'NaN' values? If it ...
AnonymousMe's user avatar
1 vote
1 answer
266 views

How to find the probability of a word to belong to a dataset of text

I have two text datasets, one with people that have a certain medical condition and another of random patients. I want to figure out what words are more likely to show up in the dataset with that ...
Kevin's user avatar
  • 261
1 vote
1 answer
25 views

Performance measurement of an event extraction system

I have developed an event extraction system from text documents. It first clusters the data corpus and extracts answers for what, when and where questions. Final answers are determined by using a ...
Nilani Algiriyage's user avatar
0 votes
1 answer
1k views

ValueError: Layer model expects 2 input(s), but it received 3 input tensors using generator

I am trying to fit a model using generator function and I get the following error: ...
manix velu's user avatar
4 votes
1 answer
73 views

encoding of text data in NLP

I'm getting data using web scraping to create a dataset. I have a 'company' column that contains the names of the companies. I would like to encode this column but i don't know how to find the ...
Lydia's user avatar
  • 43
2 votes
1 answer
1k views

Cluster words into groups of similar meaning (synonyms)

How can words be clustered into groups of similar meaning (synonyms)? I started with pre-trained word embeddings (e.g., Google News), which is great, but not perfect - a limitation arises because the ...
Ben's user avatar
  • 141
0 votes
0 answers
48 views

Effective way to find similarity between utterance(short text) and question(long text)

The challenge I have is a bunch of questions(long text) that are closely matching with an user utterance (short text). I have tried cosine similarity & Tf-iDF, BM25,Jaccard similarity, etc., but ...
Bharadwaj's user avatar
1 vote
1 answer
368 views

Advantages of CNN vs. LSTM for sequence data like text or log-files

When do you tend to use CNN rather than LSTM (or the other way round) in classification or generation tasks of sequential data like text or log-data? What are the reasons for the decision and what ...
moooo112's user avatar