Questions tagged [text]
Text is a type of data often used in data science projects involving natural language processing.
66
questions with no upvoted or accepted answers
6
votes
2
answers
129
views
Data transformations in hierarchical classification
I am building a hierarchical text classifier using the Local Classifier Per Parent Node (LCPN) approach with the 'siblings' policy
as described in the A survey of hierarchical classification across ...
- 161
6
votes
1
answer
2k
views
What is the minimum number of times a word needs to appear in word2vec training corpus for quality results?
When training a word2vec model with, eg, gensim, you can specify the minimum times a word needs to be seen (with the parameter min_count). The default value for this seems to be 5.
Are there any ...
- 203
3
votes
2
answers
358
views
NLP text representation techniques that preserve word order in sentence?
I see people are talking mostly about bag-of-words, td-idf and word embeddings. But these are at word levels. BoW and tf-idf fail to represent word orders, and word embeddings are not meant to ...
- 411
3
votes
1
answer
42
views
Text vectorizer that capture feature offset in the text?
I'm using sklearn Tfifdfvectorizer to extract feature from text towards text classification.
I believe the information I need tends to be in the beginning of the document, so I would like to somehow ...
- 53
3
votes
2
answers
2k
views
How to extract and classify data from a column in excel?
I have a column in an Excel sheet that contains a lot of data separated by || delimiters. The data can be classified to some classes like Entity, IFSC codes, ...
- 31
2
votes
1
answer
48
views
Classifying short strings of text with additional context
I have a list of short strings each identifying a city. Misspellings are very common. The example below shows some of these short strings, along with the correct city they're supposed to match.
...
- 165
2
votes
2
answers
1k
views
Clustering mixed data types - numeric, categorical, arrays, and text
I have a dataset with 4 types of data columns:
...
- 121
2
votes
1
answer
320
views
How to implement LSTM using Doc2Vec vectors to get representation?
Hi all. I'm a newbie in ML. I read and found a paper about A Multi-Level Plagiarism Detection System Based on Deep Learning Algorithms and want to implement this model . But I can't find more about ...
2
votes
1
answer
351
views
Needed: Java library to calculate text readability/complexity
In principle the same as this but for Java (and ideally for multiple languages) (e.g. flesch reading ease, smog index, flesch kincaid grade, coleman liau index, automated readability index, dale chall ...
- 294
1
vote
0
answers
18
views
Clustering Similar Articles Using Mixed Data: Seeking Advice and Validation
Question:
I'm working on a project where I need to cluster a dataset of articles based on various features, including text, numeric values, and categorical data. I've implemented a clustering approach ...
- 11
1
vote
1
answer
66
views
Dictionary-based text analysis- dealing with length
I am working on an analysis using a dictionary-based text-as-data approach. I have a dataset of texts (n=1200), and I am applying a dictionary of 50 words (I tokenize the text with each word being one ...
1
vote
1
answer
438
views
How to down\up sample text?
I have data set of 5566 samples - one column is the text of the recipe description and the other is what tax class is it.
I wish to make a classifier that would classify receipts using ML only.
I have ...
- 23
1
vote
1
answer
28
views
Partially labelled open-class classification problem with heavy overlap
Let's say we have a corpus of text, including discussions about movies and about sports. Unsupervised clustering would typically cluster into the two topics of discussion. However, we are interested ...
1
vote
2
answers
134
views
Topic modelling on long documents: intra document clustering first
I have a collection (around 1000) of very noisy, similar documents, that are each very long (>10 pages - 600 paragraphs) with multiple subsections - I want to perform topic modelling across the ...
1
vote
1
answer
25
views
Model to detect specific semantic content without labeled data
I want to build a model that can detect sentences that discuss requests for communication - like 'email me', 'phone us', 'contact us', etc. However, I do not have any labeled data which I can use to ...
- 149
1
vote
0
answers
26
views
NLP Classification labels have many similarirites, converge to and replace to only have one
I been trying to use the fuzzywuzzy library in Python to find the percentage similarity between strings in the labels. The problem I am having is that there is still many strings that are really ...
- 143
1
vote
0
answers
249
views
How can I get a value of context vector in GPT?
I'm a newbie in NLP and I'm now stuck in GPT.
The question I'm struggling with is related to a term 'context vector'
It says in the following (sorry that the material provided is written in korean)
...
- 11
1
vote
0
answers
148
views
Paragraph extraction from text
I am trying to separate scanned pages of a 3 column book into paragraphs. On the pages there can be images located in an arbitrary location, occupying part of one, two or all three of the columns.
...
- 23
1
vote
0
answers
106
views
Text data distributions comparison
I would like to know what is the best method to compare the different text data distributions. I am working on text classification. I built a model using the old dataset. Now, I would like to know, ...
- 286
1
vote
1
answer
216
views
Computer science corpus for training a language model
I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it.
Is there anything out-of-the box that I ...
- 31
1
vote
0
answers
37
views
Multimodal end-to-end deep learning
I'm thinking of working on a project that involves multiple models of data and wanted to share my thoughts to get some feedback. Think of problem of sentiment classification where the input contains ...
- 111
1
vote
1
answer
34
views
Preprocessing text so that two words without a separating space (or hyphen separated) are detected
Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the ...
- 21
1
vote
0
answers
770
views
how to resize image without changing DPI in opencv for detecting text and feeding into OCR?
i resized the image using open cv and it changed the dpi of the image from 300 dpi to 90 dpi . What is the correct way to resize image without changing its dpi in open cv . if we feed the resized ...
- 1,471
1
vote
0
answers
58
views
How do I identify specific parts of a PDF document?
I have a bunch of medical records that I have to input manually. I would like to automate this but all of the records are in different formats. What is the best strategy to build a deep learning model ...
- 191
1
vote
1
answer
1k
views
How to match a word from column and compare with other column in pandas dataframe
I have the below dataframe
...
- 111
1
vote
0
answers
56
views
Fuzzy matching of author names
We are trying to figure out what is the best approach for us to train a ML model to identify Authors. We have structured metadata of authors (given name, surname. etc) and the task is to train a model ...
- 11
1
vote
0
answers
95
views
python - Identify variable in similar sentences
I'm looking to solve the following problem: I have a list of similar sentences as my dataset, and I want to be able to type a new sentence, which is also similar to the sentences in my dataset and ...
- 111
1
vote
0
answers
59
views
Word classification in the context
I'm trying to solve a 'negation-like' classification problem, where I need to classify whether a certain word within the context has negative or positive label.
For example, how to identify whether a ...
- 503
1
vote
0
answers
17
views
Ordering quotes in a list based on user input and text analysis
A little bit of context:
I have a website that has many quotes. These quotes are organized automatically by Solr into lists of quotes, so e.g. there is a list called 'Smart Quotes' that includes ...
- 11
1
vote
0
answers
644
views
Multiclass classification with many classes and wide range of sample sizes
I'm working on a free text classification problem with over 100 classes in the training data. There is huge variation in the sample sizes of the classes: ranging from 1 to around 6000. I am using a ...
- 11
1
vote
0
answers
1k
views
Methods for string classifications
I have a list of some 100 millions of strings, each of different length. Examples:
nsdgnlnesef ngmrlxkvgrmksefsfnlj <...
- 171
1
vote
0
answers
380
views
Section/Topic segmentation in HTML and plaintext documents
GROBID (https://github.com/kermitt2/grobid) is terrific for fine-grained (section, chapter) segmentation of PDF files, but I need to do the equivalent for HTML and plaintext files.
I've tried ...
- 111
1
vote
0
answers
224
views
Generating a text training dataset from a grammar
I want to generate documents based on a grammar to build a custom training database. What are the tools and techniques to generate random texts based on a given grammar.
More specifically, I would ...
- 513
0
votes
0
answers
17
views
Handwritten text to digital versions - software, libraries, options
What types of (presumably machine learning) software/libraries exist for taking handwritten text in tables into a digital format? The tables may not always be the same. So I assume it might be fairly ...
- 101
0
votes
0
answers
19
views
Regarding TextVectorization reserved tokens
From the documentation of TextVectorization:
max_tokens: Maximum size of the vocabulary for this layer. This should
only be specified when adapting a vocabulary or when setting
pad_to_max_tokens=...
- 105
0
votes
0
answers
15
views
Do any text tagging NLP tools use a relational database or anything other than text files to annotate texts?
I am learning about how production systems implement text tagging. MILA has this XML format, and according to ChatGPT, Brat has a text file format, though haven't found much examples of that other ...
- 273
0
votes
0
answers
27
views
Why did the Double Metaphone algorithm choose to substitute and merge consonants?
Is there any literature describing the decisions behind why the mappings from input text characters to Metaphone hash consonants were made? Why did they choose to leave out vowels? Why did they merge ...
- 273
0
votes
0
answers
12
views
Highly unbalnced text data giving very low matrics
I have an unbalanced multi-class banking text data with around 76 classes. Classes are badly distributed such as one class which is combination of 240 other different categories, represents 50% of ...
- 1
0
votes
0
answers
81
views
How to input or estimate missing text data?
My task requires 32 different columns with 25 beeing independent text data.
Deleting nan values, or cutting columns which has less then 20% (of non NaNs) results in reducing dataset to less then 10% ...
- 1
0
votes
0
answers
53
views
What dimensional reduction and similarity score work for sentence embeddings created using sentence transformers
I am clustering sentence embeddings for log files, and find anomalies.
So, when I create sentence embeddings for logs using sentence transformers.
It will create vector of fixed length, which somehow ...
- 33
0
votes
0
answers
102
views
Best practice for fine tuning LLM
I have a dataset that i have collect for specific topic
the dataset is in these format :
raw text (similar to shake spare dataset) where it has no label or input, just text
Question and answer ...
- 195
0
votes
1
answer
22
views
Trouble Loading Lines from Text File with Various Encodings
I have been facing difficulties while loading specific lines from a text file. The lines contain characters such as
ٹام بیمار ÛÛ’Û”
ٹام بیمار ÛÛ’.
I have tried using different ...
0
votes
0
answers
22
views
Argument classification based on a given claim
I'm working on an nlp school project in which I have to build a model that takes a text and a claim and gives as an output whether the text is supporting or opposing the claim . .
the data that I have ...
- 101
0
votes
0
answers
20
views
Can this task for phrases be called lemmatization?
I want to 'lemmatize' phrases to dictionary entries. For instance, the following collocates can be standardized to the idiom in the aforementioned link ...
- 516
0
votes
0
answers
96
views
Creating variations of prompts for ChatGPT
I am developing a fine tune model to emulate a tech support chatbot based on my given information. I am struggling to create a large dataset (aiming for 1000 prompt/completion pairs), does anyone have ...
- 1
0
votes
0
answers
17
views
Text analysis - "interpret" bag of words
I wanted to try to create a machine learning tarot card reader. Setting up a simulated draw of tarot cards and their associated meanings is straight-forward enough. However, the listed applications I ...
- 101
0
votes
1
answer
29
views
Using relative or absolute frequencies to estimate group differences in texts
My objective is to estimate differences between how five political parties use moral words in their tweets and speeches. To that end, I have a dictionary that I pass to each tweet text / audio ...
0
votes
2
answers
292
views
Keywords extraction for business rule text classification
I would like to classify texts without using any ML model. My idea is to find a list of keywords that I would assign to each class. Then when I need to classify a new text, I can compare it with my ...
- 113
0
votes
0
answers
300
views
Text classification length
I have a set of text examples I need to learn as class A, and they are of varying lengths, say 10 sentences to 1 sentence long. I have to parse a document to find those strings of text that match one ...
- 115
0
votes
0
answers
158
views
NLP conversation data - Pre-processing steps
I have text data, so data that has been transcribed from conversations from employees to customers. So each call has a recording that has been transcribed to text.
I am looking to do some analytics ...
