Questions tagged [text]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
2answers
14 views

Need for Dense layer in Text Classifcation

While creating a model for text classification, what is the need for a Dense Layer? I noticed in multiple examples the following is the structure. A softmax is what required right instead of the Dense ...
1
vote
1answer
25 views

how to classify text based on more than one column

I passages of text to classify by topic. I am using scikit learn, e.g. linear svc, but open to other options. Currently, use only the text of each passage (column labeled ...
1
vote
1answer
19 views

Clustering mixed data types - numeric, categorical, arrays, and text

I have a dataset with 4 types of data columns: ...
0
votes
0answers
25 views

How to properly vectorise when I have several text features?

I am having several features as text and I'm using them once for a classification problem and then later for a regression problem. The textual data features themselves aren't categorical. Because as ...
0
votes
0answers
12 views

Inserting input representation at each step of LSTM

In want to train a neural net to generate lyrics based on a provided melody. For that, I have to implement a recurrent neural net (LSTM or GRU) to carry out the task. During each step of the training ...
2
votes
1answer
25 views

Are there any open-source text annotation for multi label classification tools?

I have a large texts in each document and I want to know if there are any open source text annotation tools available online for multiple label annotation. Each sentence takes two labels. If there ...
0
votes
0answers
7 views

Single numeric features row and multiple text rows corresponding to numeric data. How to classify in dataset?

For example, take a review of a product. I have numeric information corresponding to the product and also multiple reviews and I have to output a binary class for the product. How can I predict when I ...
0
votes
0answers
27 views

Machine learning model for record linkage of unstructured data

I have a list of string. Each string has position based distinct information. On the other hand, I have a totally different structure data ( say xml ). How do I match the string and the XML based on ...
0
votes
0answers
14 views

Can we use TF-IDF along with Weighted Frequency for Text Summarization?

I've been working on a text summarization problem. After studying this blog, I used weighted-frequencies of words present in a document for summarization. I would compute the weighted frequency ...
0
votes
0answers
18 views

Paragraph extraction from text

I am trying to separate scanned pages of a 3 column book into paragraphs. On the pages there can be images located in an arbitrary location, occupying part of one, two or all three of the columns. ...
0
votes
1answer
36 views

How to implement LSTM using Doc2Vec vectors to get representation?

Hi all. I'm a newbie in ML. I read and found a paper about A Multi-Level Plagiarism Detection System Based on Deep Learning Algorithms and want to implement this model . But I can't find more about ...
0
votes
0answers
9 views

Text data distributions comparison

I would like to know what is the best method to compare the different text data distributions. I am working on text classification. I built a model using the old dataset. Now, I would like to know, ...
2
votes
1answer
17 views

Text vectorizer that capture feature offset in the text?

I'm using sklearn Tfifdfvectorizer to extract feature from text towards text classification. I believe the information I need tends to be in the beginning of the document, so I would like to somehow ...
0
votes
1answer
20 views

What is the best way to encode an arbitrary collection of strings into int categorical variables?

I have a bunch of categorical labels which I want to transform into int categorical features for an ML algorithm. The problem is I don't have a prior list of the categories, so that I can't just ...
1
vote
1answer
27 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
1
vote
0answers
15 views

Multimodal end-to-end deep learning

I'm thinking of working on a project that involves multiple models of data and wanted to share my thoughts to get some feedback. Think of problem of sentiment classification where the input contains ...
0
votes
0answers
14 views

Preprocessing text so that two words without a separating space (or hyphen separated) are detected

Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the ...
0
votes
0answers
38 views

Needed: Java library to calculate text readability/complexity

In principle the same as this but for Java (and ideally for multiple languages) (e.g. flesch reading ease, smog index, flesch kincaid grade, coleman liau index, automated readability index, dale chall ...
3
votes
1answer
41 views

Classifying one particular class of documents from the rest

I am trying to build a classifier that would classify if a document is a document about sports or not. I have enough samples of sports document to train a classifier on, however I can't imagine how I ...
2
votes
1answer
45 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
2
votes
1answer
55 views

Why do probabilities sum to one and how can I set optimal threshold level?

I am working on a text classification use case. The training data has two classes, so the XBBoostClassifier and onevsrest model is classifying the test data into either of the two classes. But my ...
1
vote
0answers
16 views

Extract editing history from Microsoft Word documents? [closed]

Is there a tool to computationally extract the editing history of a given Microsoft Word Document? I have been using Apache Tika, but can only extract the last version of the text, and meta-...
1
vote
3answers
34 views

Need some info regarding string matching algorithms?

Let me explain a scenario to better explain my question, Assume I am working in a credit-card related company in which people uploads their receipts every month, I want to check if that person bought ...
1
vote
0answers
87 views

how to resize image without changing DPI in opencv for detecting text and feeding into OCR?

i resized the image using open cv and it changed the dpi of the image from 300 dpi to 90 dpi . What is the correct way to resize image without changing its dpi in open cv . if we feed the resized ...
2
votes
1answer
42 views

What is the best approach for classifying non-English text

What would be the best approach for classifying non-English (Sinhala / Tamil) text? Currently I use Fasttext. Are there any better options? I want to classify user questions into chatbot intents. ...
5
votes
2answers
54 views

Data transformations in hierarchical classification

I am building a hierarchical text classifier using the Local Classifier Per Parent Node (LCPN) approach with the 'siblings' policy as described in the A survey of hierarchical classification across ...
0
votes
0answers
22 views

How would I define a model that computes “trainable edit distance” \ string similarity for Entity Linking

I want to compute a measure of string similarity based on "edit distance". Classic solutions for edit distance predefine the cost of each editing operations, and use a combination of atomic operations ...
2
votes
1answer
46 views

How to extract keywords from a list of URLs?

I have a bunch of URLs in a text file like- ...
1
vote
2answers
29 views

Text classification for data with multiple labels per observation

I have a dataset of tweets that has been labeled by multiple people. So the columns look something like: Tweet_ID, Coder_1_Classification, Coder_2_Classification, etc. The idea is to build a tweet ...
1
vote
0answers
22 views

PDF/Text to csv table

I have very little python experience but I have been wanting to get into data science and thought I would start with pdf/text mining. Because it's something I need right now anyway. I have a list of ...
0
votes
0answers
49 views

Looking for a good text autoencoder model (for text reconstruction)

I am looking for a simple and good model that can learn to encode and reconstruct text sentences to use it in some downstream task. I tried a tensorflow seq2seq model here, but it doesn't do ...
0
votes
0answers
6 views

Finding relevant pain points in feedbacks(open text)

I have employee feedback and need to find the appropriate pain points out of their feedback. Need help with the approach and analysis. I have provided a couple of examples below. Note: The feedbacks ...
1
vote
0answers
22 views

Embedding representation for a document?

Is averaging sentence embeddings, the right way to get representation for documents. Say I have a list of sentence embeddings representing symptoms. A data point looks like these: x|S1,S2,S3 --> Y|D1,...
1
vote
0answers
16 views

How do I identify specific parts of a PDF document?

I have a bunch of medical records that I have to input manually. I would like to automate this but all of the records are in different formats. What is the best strategy to build a deep learning model ...
1
vote
1answer
156 views

How to match a word from column and compare with other column in pandas dataframe

I have the below dataframe ...
1
vote
1answer
100 views

Technical term for using regular expressions to classify text?

Background I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext. He wants to classify the claims as either having been "approved", "denied", ...
0
votes
1answer
118 views

Is there any library available for balancing imbalanced text dataset?

I have a text dataset similar to newsgroup dataset, the problem with the dataset is that it is highly imbalanced. So is there any readily built library that will do upsampling or downsampling with a ...
0
votes
1answer
128 views

Do word embeddings help with out of vocab tokens?

I am performing sentiment analysis on a custom dataset of text with Keras but am a little confused about word embeddings. I have been able to train an "Embedding" layer and have also learned to load ...
0
votes
1answer
39 views

How to utilize dictionary data set for text classification?

I have a dataset similar to newsgroup20 for classification. With the training dataset, I have a dictionary data set that explains some jargons in the training dataset. These both are different data ...
3
votes
2answers
76 views

Multiclass classification of textual data

I have a problem statement in which I have to classify the text data into various classes, but the training data is very less (250-300 data points for 4 classes). I am confused about what approach to ...
0
votes
1answer
2k views

nltk's stopwords returns “TypeError: argument of type 'LazyCorpusLoader' is not iterable”

While trying to remove stopwords using the nltk package, the following error occurred: ...
-1
votes
1answer
19 views

Identifying specific words in text

Let's say I have the following text" Is that another kitten playing in the shoes in the top right? I would like my code to extract kitten from that text. Is ...
1
vote
4answers
144 views

Is it OK to train a binary classifier using all the extremely imbalanced data if the majority class is negative?

I'm training a neural network as a binary classifier for text classification. The data is very imbalanced, where the ratio of TRUE:FALSE is approximately 100:10000 Intuitively, it feels like using ...
1
vote
1answer
18 views

Complete IPv4 Address Space

I know this was asked before, but I didn't get a definitive answer. I am trying to download a simple 17GB .txt file that contains the entire IPv4 address space. There must be a file like this already ...
0
votes
1answer
65 views

Complete IPv4 Address List

Is it possible to pull a complete list of all IPv4 addresses and put them into a text file? Since there are 4,294,967,296 IPv4 addresses and each one takes about 4 bytes I would guess the file would ...
0
votes
1answer
859 views

Extracting structure and content from invoices

Lately, I have been largely inspired by this https://rossum.ai/, which is able to extract text from invoice documents. Do you have any ideas on how this could be implemented? It's clear that they ...
3
votes
2answers
942 views

Bidirectional Encoder Representations from Transformers in R

Can anybody suggest to me, where I can find example code for R language for BERT neural network for text mining tasks. All I can see are python examples, and I need R. https://github.com/google-...
1
vote
0answers
30 views

Fuzzy matching of author names

We are trying to figure out what is the best approach for us to train a ML model to identify Authors. We have structured metadata of authors (given name, surname. etc) and the task is to train a model ...
2
votes
1answer
268 views

Implementing back translation as a data augmentation for text classification

Since back translation English->other language -> English seems like quite a useful data augmentation technique , I wanted to experiment with it. E.g. it occurred to me that languages from very ...
1
vote
2answers
97 views

Input data of variable length - two scenarios

I'm trying to figure out how I could train a neural network with inputs that have variable length. This issue comes up in the following 2 scenarios I'm trying to solve. Scenario 1: I have a long list ...