Questions tagged [text]

Text is a type of data often used in data science projects involving natural language processing.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

Fundamental difference between Tokenizer and TextVectorization in Keras

Is there any fundamental difference between Tokenizer at the word level and the TextVectorization in Keras?
0
votes
0answers
17 views

Text Classification using Language models like Bert, XLnet for 10 million data point [closed]

I am trying to train a Bert/ XLnet for the Classification task. The problem lies in the fact that training and inferencing speed is extremely slow. For a dataset size of a million, it takes more than ...
0
votes
1answer
29 views

Word embedding for a single word

I want to tackle email similarity with a word embedding approach, not an expert in embedding text but it is possible to embed emails where similar emails have similar vectors? is that okay?. I know ...
-1
votes
0answers
11 views

What word2vec representation has that make Convolutional Neural Networks perform with good results? [closed]

I'm doing a text classification task(document-level unit of analysis) and I compared the performance of word2vec(pre-trained and train over my own data) over vanilla RNN(one-hot-encoded vectors) and ...
-1
votes
0answers
12 views

How to remove irrelevant text data from a large dataset

As title says, I am working on a ML project where data were coming from a social media, and the topic about the data should be depression under Covid-19. However, when I read some of the data ...
3
votes
1answer
19 views

Extract date/duration from text

The text and output to be extracted are similar to the following : "Check it every two weeks" - two weeks "Check it on day 1 and day 14" - day 1 and day 14 "day 19 and day ...
0
votes
0answers
133 views

Excel error may be caused by pandas writing or big data? advise needed

I am reading multiple xml files extracting some data then forming a pandas Dataframe with my data. These are the main steps that I do: open an xml file extract some elements create a pandas dataframe ...
-1
votes
1answer
22 views

How to specify a location for text in a graph plotted by Python?

I would like to plot a graph of actual and predicted values with Python after doing a regression. I used the following codes. However, the text "R^2=0.91" is placed on the right hand side ...
1
vote
1answer
15 views

Which phrase should be returned in case of multiple matches when comparing text?

I want to compare one sentence to some other sentences using the Bag of Words model. Suppose that my comparing sentence is: I am playing football and there are three more sentences that I want to ...
0
votes
0answers
13 views

Methods for standardizing string data with different naming conventions

Are there conventions for standardizing text that follows different naming conventions? I have a variable of owner names that include individual names and company names. The entries follow some common ...
1
vote
1answer
18 views

How to Identify Repeating Data Entries when the Repeated Entries are Spelled or Constructed Differently

I have a dataset of entries and a variable for the owner of the entry. Some of these people occur more than once. However, the names are sometimes written differently. I want to eventually be able to ...
1
vote
1answer
14 views

Model to detect specific semantic content without labeled data

I want to build a model that can detect sentences that discuss requests for communication - like 'email me', 'phone us', 'contact us', etc. However, I do not have any labeled data which I can use to ...
0
votes
0answers
9 views

What does MiniBatchKMeans fit's reassignment_ratio parameter do exactly?

I am using scikit-learn MiniBatchKMeans to do text clustering. In the constructor method, there is a parameter reassignment_ratio, which is described in the ...
0
votes
1answer
28 views

Python to clean miswritten words with repetitive letters such as “wwwwooorrrrddss” to “words”

When cleaning text-data in Python3 for NLP, are there are any 'common practices' when it comes to addressing repetitive letters in words such as "wwwwooorrds" to "words", or "...
1
vote
1answer
12 views

distribution difference between image and text

Once for the task of image captioning I've read that, the features extracted from image and text by deep networks are from two different worlds and got different distribution. My question is how is ...
1
vote
1answer
20 views

What's the difference between sequence preprocessing and text preprocessing in Keras?

In Keras, we mainly have three types of preprocessing, i.e., sequence preprocessing, text preprocessing, and image preprocessing. However, for me, I think the meanings of the word "sequence" ...
0
votes
0answers
13 views

Converting a string to a recommendation type string

I am trying to build a recommendation system and some of the labels are ...
0
votes
1answer
127 views

Clustering Tweet Data using DBSCAN Algorithm

I am doing a tweet clustering using DBSCAN algorithm. I use all the preprocessing steps and convert sentences to vector format before applying the algorithm. ...
1
vote
0answers
20 views

NLP Classification labels have many similarirites, converge to and replace to only have one

I been trying to use the fuzzywuzzy library in Python to find the percentage similarity between strings in the labels. The problem I am having is that there is still many strings that are really ...
0
votes
3answers
41 views

Distinguish randomly generated texts from reasonable for human texts [closed]

I have strings short texts of 2 types: '23jd2032n0d2mn', 'fn830n30rn83', 'fhui29n4ok', 'qn4foml', ... and ...
1
vote
0answers
36 views

How can I get a value of context vector in GPT?

I'm a newbie in NLP and I'm now stuck in GPT. The question I'm struggling with is related to a term 'context vector' It says in the following (sorry that the material provided is written in korean) ...
1
vote
2answers
28 views

Chinese word segmentation using neural networks

Chinese text uses a character set containing tens of thousands of characters. Words in Chinese are most commonly made up of 1, 2 or 3 characters. There are no spaces or other markers between words in ...
0
votes
0answers
10 views

How to create a Question bank out of several Question papers?

I have many questions papers. I want to extract all the questions out of them so as to create a question bank. The issue is that all the question papers are not in the same format. I tried using ...
0
votes
0answers
16 views

Classification of text articles found in different PDFs

I am trying to figure out the approach to connect articles present in different pages or PDFs. The historical data that I have are: Text Articles (broken down in headlines, body text, body headline ...
0
votes
2answers
39 views

Need for Dense layer in Text Classifcation

While creating a model for text classification, what is the need for a Dense Layer? I noticed in multiple examples the following is the structure. A softmax is what required right instead of the Dense ...
1
vote
1answer
26 views

how to classify text based on more than one column

I passages of text to classify by topic. I am using scikit learn, e.g. linear svc, but open to other options. Currently, use only the text of each passage (column labeled ...
2
votes
1answer
155 views

Clustering mixed data types - numeric, categorical, arrays, and text

I have a dataset with 4 types of data columns: ...
0
votes
0answers
30 views

How to properly vectorise when I have several text features?

I am having several features as text and I'm using them once for a classification problem and then later for a regression problem. The textual data features themselves aren't categorical. Because as ...
0
votes
0answers
14 views

Inserting input representation at each step of LSTM

In want to train a neural net to generate lyrics based on a provided melody. For that, I have to implement a recurrent neural net (LSTM or GRU) to carry out the task. During each step of the training ...
2
votes
1answer
80 views

Are there any open-source text annotation for multi label classification tools?

I have a large texts in each document and I want to know if there are any open source text annotation tools available online for multiple label annotation. Each sentence takes two labels. If there ...
0
votes
0answers
37 views

Machine learning model for record linkage of unstructured data

I have a list of string. Each string has position based distinct information. On the other hand, I have a totally different structure data ( say xml ). How do I match the string and the XML based on ...
0
votes
0answers
24 views

Can we use TF-IDF along with Weighted Frequency for Text Summarization?

I've been working on a text summarization problem. After studying this blog, I used weighted-frequencies of words present in a document for summarization. I would compute the weighted frequency ...
1
vote
0answers
21 views

Paragraph extraction from text

I am trying to separate scanned pages of a 3 column book into paragraphs. On the pages there can be images located in an arbitrary location, occupying part of one, two or all three of the columns. ...
1
vote
1answer
89 views

How to implement LSTM using Doc2Vec vectors to get representation?

Hi all. I'm a newbie in ML. I read and found a paper about A Multi-Level Plagiarism Detection System Based on Deep Learning Algorithms and want to implement this model . But I can't find more about ...
0
votes
0answers
16 views

Text data distributions comparison

I would like to know what is the best method to compare the different text data distributions. I am working on text classification. I built a model using the old dataset. Now, I would like to know, ...
2
votes
1answer
23 views

Text vectorizer that capture feature offset in the text?

I'm using sklearn Tfifdfvectorizer to extract feature from text towards text classification. I believe the information I need tends to be in the beginning of the document, so I would like to somehow ...
0
votes
1answer
29 views

What is the best way to encode an arbitrary collection of strings into int categorical variables?

I have a bunch of categorical labels which I want to transform into int categorical features for an ML algorithm. The problem is I don't have a prior list of the categories, so that I can't just ...
1
vote
1answer
41 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
1
vote
0answers
22 views

Multimodal end-to-end deep learning

I'm thinking of working on a project that involves multiple models of data and wanted to share my thoughts to get some feedback. Think of problem of sentiment classification where the input contains ...
0
votes
0answers
15 views

Preprocessing text so that two words without a separating space (or hyphen separated) are detected

Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the ...
2
votes
1answer
96 views

Needed: Java library to calculate text readability/complexity

In principle the same as this but for Java (and ideally for multiple languages) (e.g. flesch reading ease, smog index, flesch kincaid grade, coleman liau index, automated readability index, dale chall ...
3
votes
1answer
45 views

Classifying one particular class of documents from the rest

I am trying to build a classifier that would classify if a document is a document about sports or not. I have enough samples of sports document to train a classifier on, however I can't imagine how I ...
2
votes
1answer
50 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
2
votes
1answer
62 views

Why do probabilities sum to one and how can I set optimal threshold level?

I am working on a text classification use case. The training data has two classes, so the XBBoostClassifier and onevsrest model is classifying the test data into either of the two classes. But my ...
1
vote
0answers
49 views

Extract editing history from Microsoft Word documents? [closed]

Is there a tool to computationally extract the editing history of a given Microsoft Word Document? I have been using Apache Tika, but can only extract the last version of the text, and meta-...
1
vote
3answers
52 views

Need some info regarding string matching algorithms?

Let me explain a scenario to better explain my question, Assume I am working in a credit-card related company in which people uploads their receipts every month, I want to check if that person bought ...
1
vote
0answers
298 views

how to resize image without changing DPI in opencv for detecting text and feeding into OCR?

i resized the image using open cv and it changed the dpi of the image from 300 dpi to 90 dpi . What is the correct way to resize image without changing its dpi in open cv . if we feed the resized ...
2
votes
1answer
45 views

What is the best approach for classifying non-English text

What would be the best approach for classifying non-English (Sinhala / Tamil) text? Currently I use Fasttext. Are there any better options? I want to classify user questions into chatbot intents. ...
5
votes
2answers
61 views

Data transformations in hierarchical classification

I am building a hierarchical text classifier using the Local Classifier Per Parent Node (LCPN) approach with the 'siblings' policy as described in the A survey of hierarchical classification across ...
2
votes
1answer
227 views

How to extract keywords from a list of URLs?

I have a bunch of URLs in a text file like- ...