Questions tagged [text]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

How am I supposed to correctly tokenize my list of documents?

I am doing a text clustering exercise. I have a set of documents I would like to cluster. My input txts is a list, every element of this list is a string which ...
-1
votes
0answers
10 views

how to convert a large xml file containing multiple xml pages into python pandas? [closed]

Every xml page is separated by a \n in the large file. So i have a large xml file sometimes about 0.5gb and in it there are multiple xml pages separated by \n. imagine that i need to take this file ...
1
vote
1answer
11 views

distribution difference between image and text

Once for the task of image captioning I've read that, the features extracted from image and text by deep networks are from two different worlds and got different distribution. My question is how is ...
0
votes
0answers
11 views

Unable to parse the rest of the xml tag? [closed]

This is a sample from the xml file: "tag1 start" Zalog is a formerly independent settlement in the eastern part of the capital "tag2 start" Ljubljana "tag2 finish" in ...
1
vote
1answer
16 views

What's the difference between sequence preprocessing and text preprocessing in Keras?

In Keras, we mainly have three types of preprocessing, i.e., sequence preprocessing, text preprocessing, and image preprocessing. However, for me, I think the meanings of the word "sequence" ...
0
votes
0answers
13 views

Converting a string to a recommendation type string

I am trying to build a recommendation system and some of the labels are ...
0
votes
1answer
25 views

Clustering Tweet Data using DBSCAN Algorithm

I am doing a tweet clustering using DBSCAN algorithm. I use all the preprocessing steps and convert sentences to vector format before applying the algorithm. ...
1
vote
0answers
18 views

NLP Classification labels have many similarirites, converge to and replace to only have one

I been trying to use the fuzzywuzzy library in Python to find the percentage similarity between strings in the labels. The problem I am having is that there is still many strings that are really ...
0
votes
3answers
38 views

Distinguish randomly generated texts from reasonable for human texts [closed]

I have strings short texts of 2 types: '23jd2032n0d2mn', 'fn830n30rn83', 'fhui29n4ok', 'qn4foml', ... and ...
1
vote
0answers
31 views

How can I get a value of context vector in GPT?

I'm a newbie in NLP and I'm now stuck in GPT. The question I'm struggling with is related to a term 'context vector' It says in the following (sorry that the material provided is written in korean) ...
1
vote
2answers
19 views

Chinese word segmentation using neural networks

Chinese text uses a character set containing tens of thousands of characters. Words in Chinese are most commonly made up of 1, 2 or 3 characters. There are no spaces or other markers between words in ...
0
votes
0answers
10 views

How to create a Question bank out of several Question papers?

I have many questions papers. I want to extract all the questions out of them so as to create a question bank. The issue is that all the question papers are not in the same format. I tried using ...
0
votes
0answers
16 views

Classification of text articles found in different PDFs

I am trying to figure out the approach to connect articles present in different pages or PDFs. The historical data that I have are: Text Articles (broken down in headlines, body text, body headline ...
0
votes
2answers
16 views

Need for Dense layer in Text Classifcation

While creating a model for text classification, what is the need for a Dense Layer? I noticed in multiple examples the following is the structure. A softmax is what required right instead of the Dense ...
1
vote
1answer
26 views

how to classify text based on more than one column

I passages of text to classify by topic. I am using scikit learn, e.g. linear svc, but open to other options. Currently, use only the text of each passage (column labeled ...
2
votes
1answer
99 views

Clustering mixed data types - numeric, categorical, arrays, and text

I have a dataset with 4 types of data columns: ...
0
votes
0answers
29 views

How to properly vectorise when I have several text features?

I am having several features as text and I'm using them once for a classification problem and then later for a regression problem. The textual data features themselves aren't categorical. Because as ...
0
votes
0answers
13 views

Inserting input representation at each step of LSTM

In want to train a neural net to generate lyrics based on a provided melody. For that, I have to implement a recurrent neural net (LSTM or GRU) to carry out the task. During each step of the training ...
2
votes
1answer
41 views

Are there any open-source text annotation for multi label classification tools?

I have a large texts in each document and I want to know if there are any open source text annotation tools available online for multiple label annotation. Each sentence takes two labels. If there ...
0
votes
0answers
33 views

Machine learning model for record linkage of unstructured data

I have a list of string. Each string has position based distinct information. On the other hand, I have a totally different structure data ( say xml ). How do I match the string and the XML based on ...
0
votes
0answers
17 views

Can we use TF-IDF along with Weighted Frequency for Text Summarization?

I've been working on a text summarization problem. After studying this blog, I used weighted-frequencies of words present in a document for summarization. I would compute the weighted frequency ...
0
votes
0answers
19 views

Paragraph extraction from text

I am trying to separate scanned pages of a 3 column book into paragraphs. On the pages there can be images located in an arbitrary location, occupying part of one, two or all three of the columns. ...
0
votes
1answer
67 views

How to implement LSTM using Doc2Vec vectors to get representation?

Hi all. I'm a newbie in ML. I read and found a paper about A Multi-Level Plagiarism Detection System Based on Deep Learning Algorithms and want to implement this model . But I can't find more about ...
0
votes
0answers
13 views

Text data distributions comparison

I would like to know what is the best method to compare the different text data distributions. I am working on text classification. I built a model using the old dataset. Now, I would like to know, ...
2
votes
1answer
19 views

Text vectorizer that capture feature offset in the text?

I'm using sklearn Tfifdfvectorizer to extract feature from text towards text classification. I believe the information I need tends to be in the beginning of the document, so I would like to somehow ...
0
votes
1answer
26 views

What is the best way to encode an arbitrary collection of strings into int categorical variables?

I have a bunch of categorical labels which I want to transform into int categorical features for an ML algorithm. The problem is I don't have a prior list of the categories, so that I can't just ...
1
vote
1answer
37 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
1
vote
0answers
20 views

Multimodal end-to-end deep learning

I'm thinking of working on a project that involves multiple models of data and wanted to share my thoughts to get some feedback. Think of problem of sentiment classification where the input contains ...
0
votes
0answers
15 views

Preprocessing text so that two words without a separating space (or hyphen separated) are detected

Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the ...
2
votes
1answer
66 views

Needed: Java library to calculate text readability/complexity

In principle the same as this but for Java (and ideally for multiple languages) (e.g. flesch reading ease, smog index, flesch kincaid grade, coleman liau index, automated readability index, dale chall ...
3
votes
1answer
44 views

Classifying one particular class of documents from the rest

I am trying to build a classifier that would classify if a document is a document about sports or not. I have enough samples of sports document to train a classifier on, however I can't imagine how I ...
2
votes
1answer
49 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
2
votes
1answer
61 views

Why do probabilities sum to one and how can I set optimal threshold level?

I am working on a text classification use case. The training data has two classes, so the XBBoostClassifier and onevsrest model is classifying the test data into either of the two classes. But my ...
1
vote
0answers
26 views

Extract editing history from Microsoft Word documents? [closed]

Is there a tool to computationally extract the editing history of a given Microsoft Word Document? I have been using Apache Tika, but can only extract the last version of the text, and meta-...
1
vote
3answers
41 views

Need some info regarding string matching algorithms?

Let me explain a scenario to better explain my question, Assume I am working in a credit-card related company in which people uploads their receipts every month, I want to check if that person bought ...
1
vote
0answers
195 views

how to resize image without changing DPI in opencv for detecting text and feeding into OCR?

i resized the image using open cv and it changed the dpi of the image from 300 dpi to 90 dpi . What is the correct way to resize image without changing its dpi in open cv . if we feed the resized ...
2
votes
1answer
44 views

What is the best approach for classifying non-English text

What would be the best approach for classifying non-English (Sinhala / Tamil) text? Currently I use Fasttext. Are there any better options? I want to classify user questions into chatbot intents. ...
5
votes
2answers
58 views

Data transformations in hierarchical classification

I am building a hierarchical text classifier using the Local Classifier Per Parent Node (LCPN) approach with the 'siblings' policy as described in the A survey of hierarchical classification across ...
2
votes
1answer
116 views

How to extract keywords from a list of URLs?

I have a bunch of URLs in a text file like- ...
1
vote
2answers
40 views

Text classification for data with multiple labels per observation

I have a dataset of tweets that has been labeled by multiple people. So the columns look something like: Tweet_ID, Coder_1_Classification, Coder_2_Classification, etc. The idea is to build a tweet ...
1
vote
0answers
21 views

How do I identify specific parts of a PDF document?

I have a bunch of medical records that I have to input manually. I would like to automate this but all of the records are in different formats. What is the best strategy to build a deep learning model ...
1
vote
1answer
268 views

How to match a word from column and compare with other column in pandas dataframe

I have the below dataframe ...
1
vote
1answer
128 views

Technical term for using regular expressions to classify text?

Background I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext. He wants to classify the claims as either having been "approved", "denied", ...
1
vote
2answers
204 views

Is there any library available for balancing imbalanced text dataset?

I have a text dataset similar to newsgroup dataset, the problem with the dataset is that it is highly imbalanced. So is there any readily built library that will do upsampling or downsampling with a ...
0
votes
1answer
180 views

Do word embeddings help with out of vocab tokens?

I am performing sentiment analysis on a custom dataset of text with Keras but am a little confused about word embeddings. I have been able to train an "Embedding" layer and have also learned to load ...
0
votes
1answer
43 views

How to utilize dictionary data set for text classification?

I have a dataset similar to newsgroup20 for classification. With the training dataset, I have a dictionary data set that explains some jargons in the training dataset. These both are different data ...
3
votes
2answers
77 views

Multiclass classification of textual data

I have a problem statement in which I have to classify the text data into various classes, but the training data is very less (250-300 data points for 4 classes). I am confused about what approach to ...
0
votes
1answer
3k views

nltk's stopwords returns “TypeError: argument of type 'LazyCorpusLoader' is not iterable”

While trying to remove stopwords using the nltk package, the following error occurred: ...
-1
votes
1answer
21 views

Identifying specific words in text

Let's say I have the following text" Is that another kitten playing in the shoes in the top right? I would like my code to extract kitten from that text. Is ...
1
vote
4answers
179 views

Is it OK to train a binary classifier using all the extremely imbalanced data if the majority class is negative?

I'm training a neural network as a binary classifier for text classification. The data is very imbalanced, where the ratio of TRUE:FALSE is approximately 100:10000 Intuitively, it feels like using ...