Questions tagged [text]

Text is a type of data often used in data science projects involving natural language processing.

Filter by
Sorted by
Tagged with
0
votes
0answers
6 views

Large summarization dataset in Portuguese [closed]

I'm looking for a large (more than 100K records) corpus dataset for a summarization task in Portuguese. Something like the CNN/DailyMail dataset (https://huggingface.co/datasets/cnn_dailymail) but in ...
0
votes
0answers
4 views

What techniques are there to train custom sentence classification models with reasonable memory footprint?

We are currently working on tasks that involve user-inputted data (e.g., question-answers, short-answer-grading), with a framework that will allow them to be improved through active learning. However, ...
0
votes
0answers
22 views

Effective way to find similarity between utterance(short text) and question(long text)

The challenge I have is a bunch of questions(long text) that are closely matching with an user utterance (short text). I have tried cosine similarity & Tf-iDF, BM25,Jaccard similarity, etc., but ...
0
votes
1answer
36 views

Advantages of CNN vs. LSTM for sequence data like text or log-files

When do you tend to use CNN rather than LSTM (or the other way round) in classification or generation tasks of sequential data like text or log-data? What are the reasons for the decision and what ...
0
votes
1answer
21 views

How to identify the feature that make the model misclassifed in text classification

Hi I am working on social media financial THAI text classification, the problem with this one is the confused classes, the misclassified prediction has a pattern that consistent as a pair. and I want ...
0
votes
0answers
20 views

text elaboration with neural network non natural language

I'm trying to understand if it's possible to elaborate with the neural network 2 input strings and predict a result string. I'm doing this job manually and it's really time consuming: basically i have ...
0
votes
0answers
17 views

Use Large Existing Dataset to Extract Information From Text

What I have: I have a large dataset of documents and their data. So I have the text of about 1M documents, and I know, for example the invoice number, of each one. What I need: Is there a way to use ...
0
votes
0answers
34 views

Fundamental difference between Tokenizer and TextVectorization in Keras

Is there any fundamental difference between Tokenizer at the word level and the TextVectorization in Keras?
0
votes
1answer
40 views

Word embedding for a single word

I want to tackle email similarity with a word embedding approach, not an expert in embedding text but it is possible to embed emails where similar emails have similar vectors? is that okay?. I know ...
0
votes
1answer
25 views

How to remove irrelevant text data from a large dataset

I am working on a ML project where data were coming from a social media, and the topic about the data should be depression under Covid-19. However, when I read some of the data retrieved, I noticed ...
3
votes
1answer
27 views

Extract date/duration from text

The text and output to be extracted are similar to the following : "Check it every two weeks" - two weeks "Check it on day 1 and day 14" - day 1 and day 14 "day 19 and day ...
0
votes
0answers
708 views

Excel error may be caused by pandas writing or big data? advise needed

I am reading multiple xml files extracting some data then forming a pandas Dataframe with my data. These are the main steps that I do: open an xml file extract some elements create a pandas dataframe ...
-1
votes
1answer
22 views

How to specify a location for text in a graph plotted by Python?

I would like to plot a graph of actual and predicted values with Python after doing a regression. I used the following codes. However, the text "R^2=0.91" is placed on the right hand side ...
1
vote
1answer
15 views

Which phrase should be returned in case of multiple matches when comparing text?

I want to compare one sentence to some other sentences using the Bag of Words model. Suppose that my comparing sentence is: I am playing football and there are three more sentences that I want to ...
0
votes
0answers
13 views

Methods for standardizing string data with different naming conventions

Are there conventions for standardizing text that follows different naming conventions? I have a variable of owner names that include individual names and company names. The entries follow some common ...
1
vote
1answer
20 views

How to Identify Repeating Data Entries when the Repeated Entries are Spelled or Constructed Differently

I have a dataset of entries and a variable for the owner of the entry. Some of these people occur more than once. However, the names are sometimes written differently. I want to eventually be able to ...
1
vote
1answer
15 views

Model to detect specific semantic content without labeled data

I want to build a model that can detect sentences that discuss requests for communication - like 'email me', 'phone us', 'contact us', etc. However, I do not have any labeled data which I can use to ...
0
votes
0answers
16 views

What does MiniBatchKMeans fit's reassignment_ratio parameter do exactly?

I am using scikit-learn MiniBatchKMeans to do text clustering. In the constructor method, there is a parameter reassignment_ratio, which is described in the ...
0
votes
1answer
30 views

Python to clean miswritten words with repetitive letters such as “wwwwooorrrrddss” to “words”

When cleaning text-data in Python3 for NLP, are there are any 'common practices' when it comes to addressing repetitive letters in words such as "wwwwooorrds" to "words", or "...
1
vote
1answer
13 views

distribution difference between image and text

Once for the task of image captioning I've read that, the features extracted from image and text by deep networks are from two different worlds and got different distribution. My question is how is ...
1
vote
1answer
26 views

What's the difference between sequence preprocessing and text preprocessing in Keras?

In Keras, we mainly have three types of preprocessing, i.e., sequence preprocessing, text preprocessing, and image preprocessing. However, for me, I think the meanings of the word "sequence" ...
0
votes
0answers
13 views

Converting a string to a recommendation type string

I am trying to build a recommendation system and some of the labels are ...
0
votes
1answer
316 views

Clustering Tweet Data using DBSCAN Algorithm

I am doing a tweet clustering using DBSCAN algorithm. I use all the preprocessing steps and convert sentences to vector format before applying the algorithm. ...
1
vote
0answers
21 views

NLP Classification labels have many similarirites, converge to and replace to only have one

I been trying to use the fuzzywuzzy library in Python to find the percentage similarity between strings in the labels. The problem I am having is that there is still many strings that are really ...
0
votes
3answers
42 views

Distinguish randomly generated texts from reasonable for human texts [closed]

I have strings short texts of 2 types: '23jd2032n0d2mn', 'fn830n30rn83', 'fhui29n4ok', 'qn4foml', ... and ...
1
vote
0answers
52 views

How can I get a value of context vector in GPT?

I'm a newbie in NLP and I'm now stuck in GPT. The question I'm struggling with is related to a term 'context vector' It says in the following (sorry that the material provided is written in korean) ...
1
vote
2answers
35 views

Chinese word segmentation using neural networks

Chinese text uses a character set containing tens of thousands of characters. Words in Chinese are most commonly made up of 1, 2 or 3 characters. There are no spaces or other markers between words in ...
0
votes
0answers
10 views

How to create a Question bank out of several Question papers?

I have many questions papers. I want to extract all the questions out of them so as to create a question bank. The issue is that all the question papers are not in the same format. I tried using ...
0
votes
0answers
17 views

Classification of text articles found in different PDFs

I am trying to figure out the approach to connect articles present in different pages or PDFs. The historical data that I have are: Text Articles (broken down in headlines, body text, body headline ...
0
votes
2answers
67 views

Need for Dense layer in Text Classifcation

While creating a model for text classification, what is the need for a Dense Layer? I noticed in multiple examples the following is the structure. A softmax is what required right instead of the Dense ...
1
vote
1answer
27 views

how to classify text based on more than one column

I passages of text to classify by topic. I am using scikit learn, e.g. linear svc, but open to other options. Currently, use only the text of each passage (column labeled ...
2
votes
1answer
234 views

Clustering mixed data types - numeric, categorical, arrays, and text

I have a dataset with 4 types of data columns: ...
0
votes
0answers
30 views

How to properly vectorise when I have several text features?

I am having several features as text and I'm using them once for a classification problem and then later for a regression problem. The textual data features themselves aren't categorical. Because as ...
0
votes
0answers
17 views

Inserting input representation at each step of LSTM

In want to train a neural net to generate lyrics based on a provided melody. For that, I have to implement a recurrent neural net (LSTM or GRU) to carry out the task. During each step of the training ...
2
votes
1answer
119 views

Are there any open-source text annotation for multi label classification tools?

I have a large texts in each document and I want to know if there are any open source text annotation tools available online for multiple label annotation. Each sentence takes two labels. If there ...
1
vote
0answers
24 views

Paragraph extraction from text

I am trying to separate scanned pages of a 3 column book into paragraphs. On the pages there can be images located in an arbitrary location, occupying part of one, two or all three of the columns. ...
1
vote
1answer
115 views

How to implement LSTM using Doc2Vec vectors to get representation?

Hi all. I'm a newbie in ML. I read and found a paper about A Multi-Level Plagiarism Detection System Based on Deep Learning Algorithms and want to implement this model . But I can't find more about ...
1
vote
0answers
21 views

Text data distributions comparison

I would like to know what is the best method to compare the different text data distributions. I am working on text classification. I built a model using the old dataset. Now, I would like to know, ...
2
votes
1answer
24 views

Text vectorizer that capture feature offset in the text?

I'm using sklearn Tfifdfvectorizer to extract feature from text towards text classification. I believe the information I need tends to be in the beginning of the document, so I would like to somehow ...
0
votes
1answer
34 views

What is the best way to encode an arbitrary collection of strings into int categorical variables?

I have a bunch of categorical labels which I want to transform into int categorical features for an ML algorithm. The problem is I don't have a prior list of the categories, so that I can't just ...
1
vote
1answer
49 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
1
vote
0answers
23 views

Multimodal end-to-end deep learning

I'm thinking of working on a project that involves multiple models of data and wanted to share my thoughts to get some feedback. Think of problem of sentiment classification where the input contains ...
1
vote
1answer
20 views

Preprocessing text so that two words without a separating space (or hyphen separated) are detected

Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the ...
2
votes
1answer
118 views

Needed: Java library to calculate text readability/complexity

In principle the same as this but for Java (and ideally for multiple languages) (e.g. flesch reading ease, smog index, flesch kincaid grade, coleman liau index, automated readability index, dale chall ...
3
votes
1answer
45 views

Classifying one particular class of documents from the rest

I am trying to build a classifier that would classify if a document is a document about sports or not. I have enough samples of sports document to train a classifier on, however I can't imagine how I ...
2
votes
1answer
52 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
2
votes
1answer
65 views

Why do probabilities sum to one and how can I set optimal threshold level?

I am working on a text classification use case. The training data has two classes, so the XBBoostClassifier and onevsrest model is classifying the test data into either of the two classes. But my ...
1
vote
0answers
97 views

Extract editing history from Microsoft Word documents? [closed]

Is there a tool to computationally extract the editing history of a given Microsoft Word Document? I have been using Apache Tika, but can only extract the last version of the text, and meta-...
1
vote
3answers
57 views

Need some info regarding string matching algorithms?

Let me explain a scenario to better explain my question, Assume I am working in a credit-card related company in which people uploads their receipts every month, I want to check if that person bought ...
1
vote
0answers
377 views

how to resize image without changing DPI in opencv for detecting text and feeding into OCR?

i resized the image using open cv and it changed the dpi of the image from 300 dpi to 90 dpi . What is the correct way to resize image without changing its dpi in open cv . if we feed the resized ...