Questions tagged [text]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
15
votes
5answers
48k views

Sentence similarity prediction

I'm looking to solve the following problem: I have a set of sentences as my dataset, and I want to be able to type a new sentence, and find the sentence that the new one is the most similar to in the ...
2
votes
1answer
37 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
2
votes
1answer
41 views

Why do probabilities sum to one and how can I set optimal threshold level?

I am working on a text classification use case. The training data has two classes, so the XBBoostClassifier and onevsrest model is classifying the test data into either of the two classes. But my ...
1
vote
0answers
11 views

Extract editing history from Microsoft Word documents? [closed]

Is there a tool to computationally extract the editing history of a given Microsoft Word Document? I have been using Apache Tika, but can only extract the last version of the text, and meta-...
1
vote
1answer
15 views

Need some info regarding string matching algorithms?

Let me explain a scenario to better explain my question, Assume I am working in a credit-card related company in which people uploads their receipts every month, I want to check if that person bought ...
2
votes
2answers
381 views

How to extract and classify data from a column in excel?

I have a column in an Excel sheet that contains a lot of data separated by || delimiters. The data can be classified to some classes like Entity, IFSC codes, ...
11
votes
2answers
9k views

How do you apply SMOTE on text classification?

Synthetic Minority Oversampling Technique (SMOTE) is an oversampling technique used in an imbalanced dataset problem. So far I have an idea how to apply it on generic, structured data. But is it ...
1
vote
0answers
14 views

how to resize image without changing DPI in opencv for detecting text and feeding into OCR?

i resized the image using open cv and it changed the dpi of the image from 300 dpi to 90 dpi . What is the correct way to resize image without changing its dpi in open cv . if we feed the resized ...
1
vote
2answers
24 views

Text classification for data with multiple labels per observation

I have a dataset of tweets that has been labeled by multiple people. So the columns look something like: Tweet_ID, Coder_1_Classification, Coder_2_Classification, etc. The idea is to build a tweet ...
6
votes
1answer
36 views

Data transformations in hierarchical classification

I am building a hierarchical text classifier using the Local Classifier Per Parent Node (LCPN) approach with the 'siblings' policy as described in the PDF: E.g. if we have the classes 1.1, 1.2, 2.1, ...
0
votes
1answer
82 views

What methods to create singular content classification from inconsistent inbound info?

I am attempting to aggregate professional profile info from multiple sources, imposing a consistent taxonomy. Specifically, the current problem is how to impose a preferred taxonomy on profiles with ...
2
votes
1answer
37 views

What is the best approach for classifying non-English text

What would be the best approach for classifying non-English (Sinhala / Tamil) text? Currently I use Fasttext. Are there any better options? I want to classify user questions into chatbot intents. ...
3
votes
1answer
2k views

Doc2vec to calculate cosine similarity - absolutely inaccurate

I'm trying to modify the Doc2vec tutorial to calculate cosine similarity and take Pandas dataframes instead of .txt documents. I ...
0
votes
0answers
18 views

How would I define a model that computes “trainable edit distance” \ string similarity for Entity Linking

I want to compute a measure of string similarity based on "edit distance". Classic solutions for edit distance predefine the cost of each editing operations, and use a combination of atomic operations ...
2
votes
1answer
36 views

How to extract keywords from a list of URLs?

I have a bunch of URLs in a text file like- ...
0
votes
1answer
33 views

How to utilize dictionary data set for text classification?

I have a dataset similar to newsgroup20 for classification. With the training dataset, I have a dictionary data set that explains some jargons in the training dataset. These both are different data ...
0
votes
1answer
470 views

Extracting structure and content from invoices

Lately, I have been largely inspired by this https://rossum.ai/, which is able to extract text from invoice documents. Do you have any ideas on how this could be implemented? It's clear that they ...
1
vote
0answers
21 views

PDF/Text to csv table

I have very little python experience but I have been wanting to get into data science and thought I would start with pdf/text mining. Because it's something I need right now anyway. I have a list of ...
0
votes
0answers
19 views

Looking for a good text autoencoder model (for text reconstruction)

I am looking for a simple and good model that can learn to encode and reconstruct text sentences to use it in some downstream task. I tried a tensorflow seq2seq model here, but it doesn't do ...
0
votes
0answers
4 views

Finding relevant pain points in feedbacks(open text)

I have employee feedback and need to find the appropriate pain points out of their feedback. Need help with the approach and analysis. I have provided a couple of examples below. Note: The feedbacks ...
1
vote
0answers
21 views

Embedding representation for a document?

Is averaging sentence embeddings, the right way to get representation for documents. Say I have a list of sentence embeddings representing symptoms. A data point looks like these: x|S1,S2,S3 --> Y|D1,...
2
votes
2answers
476 views

Bidirectional Encoder Representations from Transformers in R

Can anybody suggest to me, where I can find example code for R language for BERT neural network for text mining tasks. All I can see are python examples, and I need R. https://github.com/google-...
1
vote
0answers
15 views

How do I identify specific parts of a PDF document?

I have a bunch of medical records that I have to input manually. I would like to automate this but all of the records are in different formats. What is the best strategy to build a deep learning model ...
1
vote
2answers
56 views

Input data of variable length - two scenarios

I'm trying to figure out how I could train a neural network with inputs that have variable length. This issue comes up in the following 2 scenarios I'm trying to solve. Scenario 1: I have a long list ...
1
vote
1answer
52 views

Technical term for using regular expressions to classify text?

Background I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext. He wants to classify the claims as either having been "approved", "denied", ...
0
votes
1answer
37 views

Is there any library available for balancing imbalanced text dataset?

I have a text dataset similar to newsgroup dataset, the problem with the dataset is that it is highly imbalanced. So is there any readily built library that will do upsampling or downsampling with a ...
0
votes
1answer
34 views

Do word embeddings help with out of vocab tokens?

I am performing sentiment analysis on a custom dataset of text with Keras but am a little confused about word embeddings. I have been able to train an "Embedding" layer and have also learned to load ...
0
votes
0answers
387 views

Semantic Similarity in Universal Sentence Encoder

I am currently using Universal Sentence Encoder to embed certain sentences which I would then feed to a deep learning model to do some prediction, but just to test whether the universal sentence ...
3
votes
2answers
69 views

Multiclass classification of textual data

I have a problem statement in which I have to classify the text data into various classes, but the training data is very less (250-300 data points for 4 classes). I am confused about what approach to ...
0
votes
1answer
1k views

nltk's stopwords returns “TypeError: argument of type 'LazyCorpusLoader' is not iterable”

While trying to remove stopwords using the nltk package, the following error occurred: ...
-1
votes
1answer
18 views

Identifying specific words in text

Let's say I have the following text" Is that another kitten playing in the shoes in the top right? I would like my code to extract kitten from that text. Is ...
1
vote
4answers
81 views

Is it OK to train a binary classifier using all the extremely imbalanced data if the majority class is negative?

I'm training a neural network as a binary classifier for text classification. The data is very imbalanced, where the ratio of TRUE:FALSE is approximately 100:10000 Intuitively, it feels like using ...
0
votes
0answers
39 views

Map predictions to real text

I have read the paper "Learning to Read by Spelling" by Gutpa et al. They present a method for visual text recognition without using any paired supervisory data. In chapter 4 they describe how to ...
1
vote
1answer
17 views

Complete IPv4 Address Space

I know this was asked before, but I didn't get a definitive answer. I am trying to download a simple 17GB .txt file that contains the entire IPv4 address space. There must be a file like this already ...
0
votes
1answer
51 views

Complete IPv4 Address List

Is it possible to pull a complete list of all IPv4 addresses and put them into a text file? Since there are 4,294,967,296 IPv4 addresses and each one takes about 4 bytes I would guess the file would ...
3
votes
0answers
99 views

How to implement hierarchical labeling classification?

I am currently working on task of eCommerce product name classification, so I have categories and subcategories in product data. I noticed that using subcategories as labels delivers worse results (84%...
0
votes
0answers
60 views

Make use of multiple labels in doc2vec: Setting up the data

I am trying to implement the doc2vec algorithm with a rather small sample size: ca. 120 documents with a total of 25000 unique words. My ...
2
votes
3answers
51 views

Match a two items from two different receipts

I have two different invoices or receipts. One is a Purchase order one is something like a receipt(acknowledgement). Suppose I have ordered(PO) Wine: White Wine Red Wine Rose Wine And I receive ...
1
vote
0answers
26 views

Fuzzy matching of author names

We are trying to figure out what is the best approach for us to train a ML model to identify Authors. We have structured metadata of authors (given name, surname. etc) and the task is to train a model ...
0
votes
0answers
14 views

Do double quotes, dots and commas modify the forget weights in LSTM if retained?

I am trying to implement custom NER with LSTM. In the pre processing steps is it required to remove the punctuation marks like double quotes, dots and commas? Do they add any significance if retained? ...
0
votes
0answers
212 views

Implementing back translation as a data augmentation for text classification

Since back translation English->other language -> English seems like quite a useful data augmentation technique , I wanted to experiment with it. E.g. it occurred to me that languages from very ...
2
votes
2answers
364 views

Discovering string “motifs” in python

I have millions of strings from different sources that tend to exhibit some common patterns. Is there a way to extract these common motifs? For example, in a list (of millions) that includes strings ...
1
vote
0answers
77 views

python - Identify variable in similar sentences

I'm looking to solve the following problem: I have a list of similar sentences as my dataset, and I want to be able to type a new sentence, which is also similar to the sentences in my dataset and ...
1
vote
0answers
42 views

Word classification in the context

I'm trying to solve a 'negation-like' classification problem, where I need to classify whether a certain word within the context has negative or positive label. For example, how to identify whether a ...
1
vote
0answers
18 views

Supervisory information through side output in convolutional neural network

I am trying to implement this paper https://ieeexplore.ieee.org/document/7828014 Here they have mentioned text local (edge) and global regions as supervisory information. Side output is generated ...
1
vote
0answers
65 views

What options are out there to extract text from a group of PDFs where each PDF is formatted differently but contains the general same content [closed]

Think insurance/medical forms that come from different companies. There is no standard on formatting. I am trying to extract the text based on each section of a given form. A form might have a ...
0
votes
0answers
31 views

Text segmentation based on most probable classes

I am working on a text classification problem. As training data, I have human annotated text, which was manually segmented into sections and then these sections are labeled with some class. Training ...
0
votes
1answer
44 views

Size of Output vector from AvgW2V Vectorizer is less than Size of Input data

Hi, I have been seeing this problem for quite some time. Whenever I tried vectorizing input text data though avgw2v vectorization technique. The size of vectorized data is less than the size of the ...
2
votes
1answer
83 views

CNN to many outputs

I have a dataset with 100 columns (categorial one-hot encoded) and 1 column with text data (simple sentences) and i want to build a neural network to arround 380.000 outputs labels. I have no idea ...