Questions tagged [text]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
2
votes
1answer
89 views

CNN to many outputs

I have a dataset with 100 columns (categorial one-hot encoded) and 1 column with text data (simple sentences) and i want to build a neural network to arround 380.000 outputs labels. I have no idea ...
0
votes
1answer
223 views

Multi-Class Text Classification: Doc2Vec performing very bad compared to Hashing Vector

I have a multi-class text classification problem in hand this is similar to product category mapping where we map products to its correct Category based on the text content provided. I first created ...
0
votes
2answers
163 views

Use text similarity (cosine) instead of machine learning to classify companies into industries

I am building an industry classifier. I.e. classifying companies into industries based on a company's description. Each company can only have one industry. I took 2000 companies and assigned them ...
1
vote
1answer
36 views

Can I treat text review analysis as a regression problem?

I am playing with a dataset that contains tripadvisor restaurant reviews and their labels (either 1, 2, 3, 4 or 5 stars). Initially I was thinking of using it as a classification problem, applying ...
1
vote
0answers
13 views

Ordering quotes in a list based on user input and text analysis

A little bit of context: I have a website that has many quotes. These quotes are organized automatically by Solr into lists of quotes, so e.g. there is a list called 'Smart Quotes' that includes ...
0
votes
2answers
417 views

Street address clustering?

I have a huge dataset of addresses. I have another data stream that contains addresses that I need to match against those in the original dataset. As all the addresses are user-provided, matching them ...
2
votes
1answer
63 views

Changing multiple models into 1 model

I am working for a recruitment company on developing machine learning algorithms to automatically classify job applicants as either to be interviewed or not be interveiwed. The data is highly ...
2
votes
2answers
505 views

Accuracy reduces drastically when using TruncatedSVD with hashingvector

I have around 0.8 million product description with categories. There are around 280 categories. I want to train a model with given dataset so that in future I can predict Category for the given ...
1
vote
0answers
119 views

Where can I find a dataset for long sequence text chunking? [closed]

Context: I have documents with reviews of articles that have the following structure: Introduction: a description of the review, dates and metadata that will be discarded. (avg~180 words, std~30 ...
1
vote
2answers
42 views

Grouping company information

I have 3 different datasets with company information, in all of them I have company name, but is not perfect: For example: Dataset A: Company name: Facebook Dataset B: Company name: Facebook, Inc ...
3
votes
1answer
2k views

Is there any clear tutorial for how to use AutoEncoders with text as input

I have a pandas dataframe that describes some fields of the register. I have used one hot encoding to encode the feature vectors that are not numbers. Finally my dataset now has 4000 rows * 4 columns. ...
2
votes
0answers
677 views

What is the minimum number of times a word needs to appear in word2vec training corpus for quality results?

When training a word2vec model with, eg, gensim, you can specify the minimum times a word needs to be seen (with the parameter min_count). The default value for this seems to be 5. Are there any ...
2
votes
1answer
203 views

Why is spam detection a classification problem and not a class modelling problem

Trying to get my feet wet with machine learning on text. The most common dataset I've seen in this space is the sms dataset with classes ham and spam. And the most common and successful approach ...
2
votes
1answer
4k views

One hot encoding at character level with Keras

I am reading Chollet's book on deep learning at the moment and in the NLP chapter he says: ...
3
votes
1answer
4k views

How does ,the Mutlinomial Bayes's alpha parameter, affects the text classification task?

I would like to know how the alpha parameter, in Multinomial Bayes, affects the text classification task. I know that this parameter is correlated to the algorithm'...
6
votes
1answer
5k views

Which type auto encoder gives best results for text

I did I couple of examples for auto encoders for images and they worked fine. Now I want to do an auto encoder for text that takes as input a sentence and returns the same sentence. But when I try to ...
1
vote
0answers
84 views

Categorize text as Body, Heading in a loosely formatted document

Given a semi-structured document with only texts and images, and some style properties present on the text, What is the best possible way to classify the text present in the document to any formal ...
11
votes
2answers
10k views

How do you apply SMOTE on text classification?

Synthetic Minority Oversampling Technique (SMOTE) is an oversampling technique used in an imbalanced dataset problem. So far I have an idea how to apply it on generic, structured data. But is it ...
1
vote
0answers
549 views

Multiclass classification with many classes and wide range of sample sizes

I'm working on a free text classification problem with over 100 classes in the training data. There is huge variation in the sample sizes of the classes: ranging from 1 to around 6000. I am using a ...
1
vote
1answer
524 views

Unsupervised clustering of unstructured text by document type

I have 100,000+ PDF healthcare documents from which I have extracted text. I would like to cluster these documents by type (e.g. pathology report, doctor visit notes, prescription orders, etc.) The ...
0
votes
2answers
52 views

Classify text labels in to a similar category [closed]

I'm trying to classify same kind of text labels in to one category. For example, if I have labels like qty, quantity, qty_no all of them should direct to Quantity. ...
1
vote
2answers
355 views

Text classification problem using Python or R

I am a novice in machine learning and new to NLP. I am looking for ideas on how to solve the below two problems. I have a dataset with two columns, "Titles" and "Description". Titles column has names ...
1
vote
3answers
4k views

Grouping of similar looking text

I have a data frame which has two columns, "Title" and "Description". The title column has a bunch of titles related to clinical lab tests. Unfortunately, most of the titles are a repeat of the same ...
1
vote
1answer
115 views

How to add incorporate meta data into text classification? [closed]

I have a collection of statements which I need to classify into 5 classes. Each statement have meta data in different columns: Author|Editor| date of release| statement | Class How can one use the ...
3
votes
1answer
2k views

Doc2vec to calculate cosine similarity - absolutely inaccurate

I'm trying to modify the Doc2vec tutorial to calculate cosine similarity and take Pandas dataframes instead of .txt documents. I ...
-3
votes
1answer
34 views

Algorithms/services to know an “iPhone case” is not an “iPhone”, in the context of complex item descriptions? [closed]

We are trying to implement a highly accurate search, based on user-entered search terms, into a large product database. For example, if the user searches for "iPhone", then one of these is ...
15
votes
5answers
51k views

Sentence similarity prediction

I'm looking to solve the following problem: I have a set of sentences as my dataset, and I want to be able to type a new sentence, and find the sentence that the new one is the most similar to in the ...
2
votes
2answers
376 views

Discovering string “motifs” in python

I have millions of strings from different sources that tend to exhibit some common patterns. Is there a way to extract these common motifs? For example, in a list (of millions) that includes strings ...
1
vote
1answer
146 views

Where can I find datasets with labeled duplicate text documents?

I'm working on detecting duplicate text documents using a classifier. I am looking for training data - a corpus of text documents and corresponding metadata which lists out pairs of duplicate ...
-1
votes
1answer
74 views

Text standardisation for manually entered data

I am working on a project that involves dealing with manually entered text data. I have a dataset of customs records where the customs officers manually enter the name and address of companies ...
0
votes
1answer
84 views

What methods to create singular content classification from inconsistent inbound info?

I am attempting to aggregate professional profile info from multiple sources, imposing a consistent taxonomy. Specifically, the current problem is how to impose a preferred taxonomy on profiles with ...
-1
votes
1answer
263 views

ML project ideas for dataset [closed]

Not sure if this is the right forum, but currently i have a dataset which contains a list of TV shows. Each record contains pricing between competitors (price in provider 1. Example: Itunes) TV show ...
1
vote
0answers
570 views

Methods for string classifications

I have a list of some 100 millions of strings, each of different length. Examples: nsdgnlnesef ngmrlxkvgrmksefsfnlj <...
0
votes
3answers
340 views

Using training data generated with pure regular expressions - Can machine learning surpass the accuracy of your regular expression?

For text classification with machine learning - If your training data was generated purely with regular expressions, is it possible to train a machine learning model with this training data which will ...
2
votes
1answer
2k views

What are some function/package in R to find similarity of individual words not in the context of sentences?

what are some function/package in R/python to find similarity of individual words not in the context of sentences? As a novice, I searched and found methods like adist, cosine similarity, word2vec, ...
7
votes
1answer
4k views

How to use TFIDF vectors with multinomial naive bayes?

Say we have used the TFIDF transform to encode documents into continuous-valued features. How would we now use this as input to a Naive Bayes classifier? Bernoulli naive-bayes is out, because our ...
0
votes
3answers
2k views

How to use binary text classifier(built using SVM with TF-IDF) to classify new text document?

I have built binary text classifier using SVM on TF-IDF for news articles(Sports: Non-Sports). But I not sure how to classify new document using this model. Since TF-IDF is calculated based on the ...
1
vote
0answers
349 views

Section/Topic segmentation in HTML and plaintext documents

GROBID (https://github.com/kermitt2/grobid) is terrific for fine-grained (section, chapter) segmentation of PDF files, but I need to do the equivalent for HTML and plaintext files. I've tried ...
1
vote
1answer
569 views

Text processing

I am completely new to analyze cluster texts, I'm using Goodreads API to get Books synopsis. My goal is to group similar books, for example: Politics Music Biographies etc... While Goodreads provide ...
5
votes
2answers
7k views

Text similarity using RNN

Data set contains records of short text, typically a sentence. The goal is to find duplicated records and similar records. Currently, I have tried R package 'text2vec', the glove word vectors and the ...
0
votes
1answer
141 views

Text Mining of Research Paper Abstracts

I'm trying to build a system which tackles the following problem. Given a list of research papers L, I would like to extract the following information from each paper: The problem addressed by the ...
-1
votes
1answer
52 views

What is the best solution to find the readability of texts?

I want to find the readability of texts. Can I use a classification method for that? For example, by collecting some basic, intermediate, and advanced texts as training sets and then finding the ...
0
votes
1answer
632 views

Which approach for user classification on chat text (classifier, representation, features)?

I'm trying to train a classifier to classify text from a chat between 2 users so later on I can predict who of the two users is more likely to say X sentence/word. To get there I mined the text from ...
4
votes
4answers
185 views

Extract 2 pieces of information from a string - what to use?

First of all, I am a complete newbie in regard to data science and I am not asking for the complete solution but some guidance as to what I should read up to achieve my task (what algorithms, ...
1
vote
2answers
2k views

Text similarities

I have a list of E-mails, and I'm trying to find "similar" E-mails. For example: DanB@gmail.com DanB1@gmail.com Dan12B@yahoo.com will be "similar" for me. I'll be happy to hear if ...
3
votes
1answer
3k views

Choosing the right parameters to train a Tf-Idf vectoriser

I'm very new to the DS world, so please bear with my ignorance. I'm trying to analyse user comments in Spanish. I have a somewhat small dataset (in the 100k's -- is that small?), and when I run the ...
1
vote
0answers
197 views

Generating a text training dataset from a grammar

I want to generate documents based on a grammar to build a custom training database. What are the tools and techniques to generate random texts based on a given grammar. More specifically, I would ...
0
votes
1answer
1k views

How to down-weight non-words in text classification?

Background: Documents coming in as well as training set have gone through Apache Tika with Tesseract for inline images. This works well, except when it doesn't. Many docs are old, scanned images ...
4
votes
3answers
903 views

How can I group texts with similar content together?

I need to find a solution to group a corpus of texts according to document similarity. Premising I have no experience in ML - only a few readings - I'd like to know if calculating the tf-idf on each ...
3
votes
1answer
110 views

How to classify support call texts?

I have a spreadsheet with thousands of records regarding support requests. Case number, Issue description, etc. Our goal is to classify these records in many categories in order to assign them the ...