Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [text]

The tag has no usage guidance.

0
votes
1answer
33 views

Complete IPv4 Address List

Is it possible to pull a complete list of all IPv4 addresses and put them into a text file? Since there are 4,294,967,296 IPv4 addresses and each one takes about 4 bytes I would guess the file would ...
0
votes
1answer
10 views

Extracting structure and content from invoices

I have been lately hugely inspired https://rossum.ai/, which is able to extract text from invoice documents. Do you have any ideas how this could be implemented? Well, it's clear that they did a lot ...
0
votes
0answers
13 views

Make use of multiple labels in doc2vec: Setting up the data

I am trying to implement the doc2vec algorithm with a rather small sample size: ca. 120 documents with a total of 25000 unique words. My ...
2
votes
1answer
12 views

Bidirectional Encoder Representations from Transformers in R

Can anybody suggest to me, where I can find example code for R language for BERT neural network for text mining tasks. All I can see are python examples, and I need R. https://github.com/google-...
0
votes
0answers
19 views

Naive bayes text analysis in weka, in production

I've created java code to train weka naive bayes on a database of text. However, I don't know how to actually use the model for new text. My training instances have the attributes "id", "text", "...
1
vote
0answers
19 views

Fuzzy matching of author names

We are trying to figure out what is the best approach for us to train a ML model to identify Authors. We have structured metadata of authors (given name, surname. etc) and the task is to train a model ...
0
votes
0answers
13 views

Do double quotes, dots and commas modify the forget weights in LSTM if retained?

I am trying to implement custom NER with LSTM. In the pre processing steps is it required to remove the punctuation marks like double quotes, dots and commas? Do they add any significance if retained? ...
0
votes
0answers
44 views

Implementing back translation as a data augmentation for text classification

Since back translation English->other language -> English seems like quite a useful data augmentation technique , I wanted to experiment with it. E.g. it occurred to me that languages from very ...
1
vote
2answers
35 views

Input data of variable length - two scenarios

I'm trying to figure out how I could train a neural network with inputs that have variable length. This issue comes up in the following 2 scenarios I'm trying to solve. Scenario 1: I have a long list ...
1
vote
0answers
60 views

python - Identify variable in similar sentences

I'm looking to solve the following problem: I have a list of similar sentences as my dataset, and I want to be able to type a new sentence, which is also similar to the sentences in my dataset and ...
2
votes
3answers
37 views

Match a two items from two different receipts

I have two different invoices or receipts. One is a Purchase order one is something like a receipt(acknowledgement). Suppose I have ordered(PO) Wine: White Wine Red Wine Rose Wine And I receive ...
3
votes
0answers
37 views

How to implement hierarchical labeling classification?

I am currently working on task of eCommerce product name classification, so I have categories and subcategories in product data. I noticed that using subcategories as labels delivers worse results (84%...
2
votes
1answer
65 views

How to extract and classify data from a column in excel?

I have a column in an Excel sheet that contains a lot of data separated by || delimiters. The data can be classified to some classes like Entity, IFSC codes, ...
1
vote
0answers
23 views

Word classification in the context

I'm trying to solve a 'negation-like' classification problem, where I need to classify whether a certain word within the context has negative or positive label. For example, how to identify whether a ...
1
vote
0answers
17 views

Supervisory information through side output in convolutional neural network

I am trying to implement this paper https://ieeexplore.ieee.org/document/7828014 Here they have mentioned text local (edge) and global regions as supervisory information. Side output is generated ...
1
vote
0answers
27 views

What options are out there to extract text from a group of PDFs where each PDF is formatted differently but contains the general same content [closed]

Think insurance/medical forms that come from different companies. There is no standard on formatting. I am trying to extract the text based on each section of a given form. A form might have a ...
0
votes
0answers
19 views

Text segmentation based on most probable classes

I am working on a text classification problem. As training data, I have human annotated text, which was manually segmented into sections and then these sections are labeled with some class. Training ...
0
votes
1answer
25 views

Size of Output vector from AvgW2V Vectorizer is less than Size of Input data

Hi, I have been seeing this problem for quite some time. Whenever I tried vectorizing input text data though avgw2v vectorization technique. The size of vectorized data is less than the size of the ...
0
votes
1answer
31 views

How to use correlation matrix when the dataset contains multiple columns with text data? [closed]

How to use it with Amazon fine food reviews dataset?
2
votes
1answer
56 views

CNN to many outputs

I have a dataset with 100 columns (categorial one-hot encoded) and 1 column with text data (simple sentences) and i want to build a neural network to arround 380.000 outputs labels. I have no idea ...
0
votes
1answer
141 views

Multi-Class Text Classification: Doc2Vec performing very bad compared to Hashing Vector

I have a multi-class text classification problem in hand this is similar to product category mapping where we map products to its correct Category based on the text content provided. I first created ...
0
votes
2answers
98 views

Use text similarity (cosine) instead of machine learning to classify companies into industries

I am building an industry classifier. I.e. classifying companies into industries based on a company's description. Each company can only have one industry. I took 2000 companies and assigned them ...
1
vote
1answer
26 views

Can I treat text review analysis as a regression problem?

I am playing with a dataset that contains tripadvisor restaurant reviews and their labels (either 1, 2, 3, 4 or 5 stars). Initially I was thinking of using it as a classification problem, applying ...
1
vote
0answers
13 views

Ordering quotes in a list based on user input and text analysis

A little bit of context: I have a website that has many quotes. These quotes are organized automatically by Solr into lists of quotes, so e.g. there is a list called 'Smart Quotes' that includes ...
0
votes
2answers
211 views

Street address clustering?

I have a huge dataset of addresses. I have another data stream that contains addresses that I need to match against those in the original dataset. As all the addresses are user-provided, matching them ...
2
votes
1answer
41 views

Changing multiple models into 1 model

I am working for a recruitment company on developing machine learning algorithms to automatically classify job applicants as either to be interviewed or not be interveiwed. The data is highly ...
2
votes
2answers
271 views

Accuracy reduces drastically when using TruncatedSVD with hashingvector

I have around 0.8 million product description with categories. There are around 280 categories. I want to train a model with given dataset so that in future I can predict Category for the given ...
1
vote
0answers
66 views

Where can I find a dataset for long sequence text chunking? [closed]

Context: I have documents with reviews of articles that have the following structure: Introduction: a description of the review, dates and metadata that will be discarded. (avg~180 words, std~30 ...
1
vote
2answers
35 views

Grouping company information

I have 3 different datasets with company information, in all of them I have company name, but is not perfect: For example: Dataset A: Company name: Facebook Dataset B: Company name: Facebook, Inc ...
3
votes
1answer
989 views

Is there any clear tutorial for how to use AutoEncoders with text as input

I have a pandas dataframe that describes some fields of the register. I have used one hot encoding to encode the feature vectors that are not numbers. Finally my dataset now has 4000 rows * 4 columns. ...
2
votes
0answers
425 views

What is the minimum number of times a word needs to appear in word2vec training corpus for quality results?

When training a word2vec model with, eg, gensim, you can specify the minimum times a word needs to be seen (with the parameter min_count). The default value for this seems to be 5. Are there any ...
2
votes
1answer
135 views

Why is spam detection a classification problem and not a class modelling problem

Trying to get my feet wet with machine learning on text. The most common dataset I've seen in this space is the sms dataset with classes ham and spam. And the most common and successful approach ...
1
vote
1answer
3k views

One hot encoding at character level with Keras

I am reading Chollet's book on deep learning at the moment and in the NLP chapter he says: ...
2
votes
1answer
2k views

How does ,the Mutlinomial Bayes's alpha parameter, affects the text classification task?

I would like to know how the alpha parameter, in Multinomial Bayes, affects the text classification task. I know that this parameter is correlated to the algorithm'...
3
votes
1answer
3k views

Which type auto encoder gives best results for text

I did I couple of examples for auto encoders for images and they worked fine. Now I want to do an auto encoder for text that takes as input a sentence and returns the same sentence. But when I try to ...
1
vote
0answers
63 views

Categorize text as Body, Heading in a loosely formatted document

Given a semi-structured document with only texts and images, and some style properties present on the text, What is the best possible way to classify the text present in the document to any formal ...
7
votes
1answer
6k views

How do you apply SMOTE on text classification?

Synthetic Minority Oversampling Technique (SMOTE) is an oversampling technique used in an imbalanced dataset problem. So far I have an idea how to apply it on generic, structured data. But is it ...
1
vote
0answers
488 views

Multiclass classification with many classes and wide range of sample sizes

I'm working on a free text classification problem with over 100 classes in the training data. There is huge variation in the sample sizes of the classes: ranging from 1 to around 6000. I am using a ...
1
vote
1answer
415 views

Unsupervised clustering of unstructured text by document type

I have 100,000+ PDF healthcare documents from which I have extracted text. I would like to cluster these documents by type (e.g. pathology report, doctor visit notes, prescription orders, etc.) The ...
0
votes
2answers
48 views

Classify text labels in to a similar category [closed]

I'm trying to classify same kind of text labels in to one category. For example, if I have labels like qty, quantity, qty_no all of them should direct to Quantity. ...
1
vote
2answers
325 views

Text classification problem using Python or R

I am a novice in machine learning and new to NLP. I am looking for ideas on how to solve the below two problems. I have a dataset with two columns, "Titles" and "Description". Titles column has names ...
1
vote
3answers
3k views

Grouping of similar looking text

I have a data frame which has two columns, "Title" and "Description". The title column has a bunch of titles related to clinical lab tests. Unfortunately, most of the titles are a repeat of the same ...
1
vote
1answer
74 views

How to add incorporate meta data into text classification? [closed]

I have a collection of statements which I need to classify into 5 classes. Each statement have meta data in different columns: Author|Editor| date of release| statement | Class How can one use the ...
2
votes
0answers
2k views

Doc2vec to calculate cosine similarity - absolutely inaccurate

I'm trying to modify the Doc2vec tutorial to calculate cosine similarity and take Pandas dataframes instead of .txt documents. I ...
-3
votes
1answer
32 views

Algorithms/services to know an “iPhone case” is not an “iPhone”, in the context of complex item descriptions? [closed]

We are trying to implement a highly accurate search, based on user-entered search terms, into a large product database. For example, if the user searches for "iPhone", then one of these is ...
11
votes
4answers
31k views

Sentence similarity prediction

I'm looking to solve the following problem: I have a set of sentences as my dataset, and I want to be able to type a new sentence, and find the sentence that the new one is the most similar to in the ...
2
votes
2answers
276 views

Discovering string “motifs” in python

I have millions of strings from different sources that tend to exhibit some common patterns. Is there a way to extract these common motifs? For example, in a list (of millions) that includes strings ...
0
votes
1answer
100 views

Where can I find datasets with labeled duplicate text documents?

I'm working on detecting duplicate text documents using a classifier. I am looking for training data - a corpus of text documents and corresponding metadata which lists out pairs of duplicate ...
-1
votes
1answer
65 views

Text standardisation for manually entered data

I am working on a project that involves dealing with manually entered text data. I have a dataset of customs records where the customs officers manually enter the name and address of companies ...
0
votes
1answer
80 views

What methods to create singular content classification from inconsistent inbound info?

I am attempting to aggregate professional profile info from multiple sources, imposing a consistent taxonomy. Specifically, the current problem is how to impose a preferred taxonomy on profiles with ...