Questions tagged [text]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
3answers
29 views

Distinguish randomly generated texts from reasonable for human texts [closed]

I have strings short texts of 2 types: '23jd2032n0d2mn', 'fn830n30rn83', 'fhui29n4ok', 'qn4foml', ... and ...
1
vote
2answers
15 views

Chinese word segmentation using neural networks

Chinese text uses a character set containing tens of thousands of characters. Words in Chinese are most commonly made up of 1, 2 or 3 characters. There are no spaces or other markers between words in ...
0
votes
1answer
48 views

How to implement LSTM using Doc2Vec vectors to get representation?

Hi all. I'm a newbie in ML. I read and found a paper about A Multi-Level Plagiarism Detection System Based on Deep Learning Algorithms and want to implement this model . But I can't find more about ...
2
votes
2answers
652 views

How to extract and classify data from a column in excel?

I have a column in an Excel sheet that contains a lot of data separated by || delimiters. The data can be classified to some classes like Entity, IFSC codes, ...
0
votes
0answers
19 views

How can I get a value of context vector in GPT?

I'm a newbie in NLP and I'm now stuck in GPT. The question I'm struggling with is related to a term 'context vector' It says in the following (sorry that the material provided is written in korean) ...
2
votes
1answer
55 views

Needed: Java library to calculate text readability/complexity

In principle the same as this but for Java (and ideally for multiple languages) (e.g. flesch reading ease, smog index, flesch kincaid grade, coleman liau index, automated readability index, dale chall ...
2
votes
1answer
19 views

Text vectorizer that capture feature offset in the text?

I'm using sklearn Tfifdfvectorizer to extract feature from text towards text classification. I believe the information I need tends to be in the beginning of the document, so I would like to somehow ...
0
votes
1answer
87 views

What methods to create singular content classification from inconsistent inbound info?

I am attempting to aggregate professional profile info from multiple sources, imposing a consistent taxonomy. Specifically, the current problem is how to impose a preferred taxonomy on profiles with ...
1
vote
2answers
35 views

Text classification for data with multiple labels per observation

I have a dataset of tweets that has been labeled by multiple people. So the columns look something like: Tweet_ID, Coder_1_Classification, Coder_2_Classification, etc. The idea is to build a tweet ...
1
vote
2answers
156 views

Is there any library available for balancing imbalanced text dataset?

I have a text dataset similar to newsgroup dataset, the problem with the dataset is that it is highly imbalanced. So is there any readily built library that will do upsampling or downsampling with a ...
5
votes
2answers
57 views

Data transformations in hierarchical classification

I am building a hierarchical text classifier using the Local Classifier Per Parent Node (LCPN) approach with the 'siblings' policy as described in the A survey of hierarchical classification across ...
2
votes
1answer
32 views

Are there any open-source text annotation for multi label classification tools?

I have a large texts in each document and I want to know if there are any open source text annotation tools available online for multiple label annotation. Each sentence takes two labels. If there ...
2
votes
1answer
2k views

What are some function/package in R to find similarity of individual words not in the context of sentences?

what are some function/package in R/python to find similarity of individual words not in the context of sentences? As a novice, I searched and found methods like adist, cosine similarity, word2vec, ...
3
votes
2answers
1k views

Word analysis in Python

I have a list of documents which look like this: ...
0
votes
1answer
24 views

What is the best way to encode an arbitrary collection of strings into int categorical variables?

I have a bunch of categorical labels which I want to transform into int categorical features for an ML algorithm. The problem is I don't have a prior list of the categories, so that I can't just ...
0
votes
1answer
40 views

How to utilize dictionary data set for text classification?

I have a dataset similar to newsgroup20 for classification. With the training dataset, I have a dictionary data set that explains some jargons in the training dataset. These both are different data ...
0
votes
0answers
10 views

How to create a Question bank out of several Question papers?

I have many questions papers. I want to extract all the questions out of them so as to create a question bank. The issue is that all the question papers are not in the same format. I tried using ...
1
vote
1answer
31 views

Computer science corpus for training a language model

I am looking for a domain specific computer science corpus of at least 20M words (preferable >50M words), for the purpose of training a language model in it. Is there anything out-of-the box that I ...
0
votes
1answer
940 views

Extracting structure and content from invoices

Lately, I have been largely inspired by this https://rossum.ai/, which is able to extract text from invoice documents. Do you have any ideas on how this could be implemented? It's clear that they ...
0
votes
0answers
15 views

Classification of text articles found in different PDFs

I am trying to figure out the approach to connect articles present in different pages or PDFs. The historical data that I have are: Text Articles (broken down in headlines, body text, body headline ...
1
vote
3answers
36 views

Need some info regarding string matching algorithms?

Let me explain a scenario to better explain my question, Assume I am working in a credit-card related company in which people uploads their receipts every month, I want to check if that person bought ...
3
votes
1answer
43 views

Classifying one particular class of documents from the rest

I am trying to build a classifier that would classify if a document is a document about sports or not. I have enough samples of sports document to train a classifier on, however I can't imagine how I ...
1
vote
1answer
25 views

how to classify text based on more than one column

I passages of text to classify by topic. I am using scikit learn, e.g. linear svc, but open to other options. Currently, use only the text of each passage (column labeled ...
0
votes
2answers
16 views

Need for Dense layer in Text Classifcation

While creating a model for text classification, what is the need for a Dense Layer? I noticed in multiple examples the following is the structure. A softmax is what required right instead of the Dense ...
0
votes
0answers
27 views

How to properly vectorise when I have several text features?

I am having several features as text and I'm using them once for a classification problem and then later for a regression problem. The textual data features themselves aren't categorical. Because as ...
0
votes
0answers
13 views

Inserting input representation at each step of LSTM

In want to train a neural net to generate lyrics based on a provided melody. For that, I have to implement a recurrent neural net (LSTM or GRU) to carry out the task. During each step of the training ...
0
votes
0answers
31 views

Machine learning model for record linkage of unstructured data

I have a list of string. Each string has position based distinct information. On the other hand, I have a totally different structure data ( say xml ). How do I match the string and the XML based on ...
0
votes
0answers
7 views

Single numeric features row and multiple text rows corresponding to numeric data. How to classify in dataset?

For example, take a review of a product. I have numeric information corresponding to the product and also multiple reviews and I have to output a binary class for the product. How can I predict when I ...
2
votes
1answer
291 views

Implementing back translation as a data augmentation for text classification

Since back translation English->other language -> English seems like quite a useful data augmentation technique , I wanted to experiment with it. E.g. it occurred to me that languages from very ...
0
votes
0answers
19 views

Paragraph extraction from text

I am trying to separate scanned pages of a 3 column book into paragraphs. On the pages there can be images located in an arbitrary location, occupying part of one, two or all three of the columns. ...
0
votes
0answers
15 views

Can we use TF-IDF along with Weighted Frequency for Text Summarization?

I've been working on a text summarization problem. After studying this blog, I used weighted-frequencies of words present in a document for summarization. I would compute the weighted frequency ...
0
votes
0answers
10 views

Text data distributions comparison

I would like to know what is the best method to compare the different text data distributions. I am working on text classification. I built a model using the old dataset. Now, I would like to know, ...
4
votes
3answers
1k views

How can I group texts with similar content together?

I need to find a solution to group a corpus of texts according to document similarity. Premising I have no experience in ML - only a few readings - I'd like to know if calculating the tf-idf on each ...
1
vote
0answers
18 views

Multimodal end-to-end deep learning

I'm thinking of working on a project that involves multiple models of data and wanted to share my thoughts to get some feedback. Think of problem of sentiment classification where the input contains ...
1
vote
1answer
117 views

Technical term for using regular expressions to classify text?

Background I'm helping a researcher programmatically classify ~123,000 US Government court case files stored in plaintext. He wants to classify the claims as either having been "approved", "denied", ...
0
votes
0answers
14 views

Preprocessing text so that two words without a separating space (or hyphen separated) are detected

Let's say I have a text corpus with inconsistently written bi-grams. An example would be "bi gram", "bi-gram", "bigram". Is there any standard text preprocessing method to normalize all these as the ...
20
votes
5answers
61k views

Sentence similarity prediction

I'm looking to solve the following problem: I have a set of sentences as my dataset, and I want to be able to type a new sentence, and find the sentence that the new one is the most similar to in the ...
2
votes
1answer
48 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
2
votes
1answer
59 views

Why do probabilities sum to one and how can I set optimal threshold level?

I am working on a text classification use case. The training data has two classes, so the XBBoostClassifier and onevsrest model is classifying the test data into either of the two classes. But my ...
1
vote
0answers
20 views

Extract editing history from Microsoft Word documents? [closed]

Is there a tool to computationally extract the editing history of a given Microsoft Word Document? I have been using Apache Tika, but can only extract the last version of the text, and meta-...
16
votes
2answers
13k views

How do you apply SMOTE on text classification?

Synthetic Minority Oversampling Technique (SMOTE) is an oversampling technique used in an imbalanced dataset problem. So far I have an idea how to apply it on generic, structured data. But is it ...
1
vote
0answers
134 views

how to resize image without changing DPI in opencv for detecting text and feeding into OCR?

i resized the image using open cv and it changed the dpi of the image from 300 dpi to 90 dpi . What is the correct way to resize image without changing its dpi in open cv . if we feed the resized ...
2
votes
1answer
43 views

What is the best approach for classifying non-English text

What would be the best approach for classifying non-English (Sinhala / Tamil) text? Currently I use Fasttext. Are there any better options? I want to classify user questions into chatbot intents. ...
4
votes
1answer
3k views

Doc2vec to calculate cosine similarity - absolutely inaccurate

I'm trying to modify the Doc2vec tutorial to calculate cosine similarity and take Pandas dataframes instead of .txt documents. I ...
0
votes
0answers
26 views

How would I define a model that computes “trainable edit distance” \ string similarity for Entity Linking

I want to compute a measure of string similarity based on "edit distance". Classic solutions for edit distance predefine the cost of each editing operations, and use a combination of atomic operations ...
2
votes
1answer
60 views

How to extract keywords from a list of URLs?

I have a bunch of URLs in a text file like- ...
1
vote
0answers
22 views

PDF/Text to csv table

I have very little python experience but I have been wanting to get into data science and thought I would start with pdf/text mining. Because it's something I need right now anyway. I have a list of ...
0
votes
0answers
56 views

Looking for a good text autoencoder model (for text reconstruction)

I am looking for a simple and good model that can learn to encode and reconstruct text sentences to use it in some downstream task. I tried a tensorflow seq2seq model here, but it doesn't do ...