Questions tagged [text]

Text is a type of data often used in data science projects involving natural language processing.

Filter by
Sorted by
Tagged with
34 votes
6 answers
82k views

Sentence similarity prediction

I'm looking to solve the following problem: I have a set of sentences as my dataset, and I want to be able to type a new sentence, and find the sentence that the new one is the most similar to in the ...
  • 1,268
25 votes
3 answers
26k views

How do you apply SMOTE on text classification?

Synthetic Minority Oversampling Technique (SMOTE) is an oversampling technique used in an imbalanced dataset problem. So far I have an idea how to apply it on generic, structured data. But is it ...
  • 369
9 votes
1 answer
7k views

Which type auto encoder gives best results for text

I did I couple of examples for auto encoders for images and they worked fine. Now I want to do an auto encoder for text that takes as input a sentence and returns the same sentence. But when I try to ...
  • 109
9 votes
1 answer
5k views

How to use TFIDF vectors with multinomial naive bayes?

Say we have used the TFIDF transform to encode documents into continuous-valued features. How would we now use this as input to a Naive Bayes classifier? Bernoulli naive-bayes is out, because our ...
  • 191
9 votes
2 answers
342 views

How to implement hierarchical labeling classification?

I am currently working on the task of eCommerce product name classification, so I have categories and subcategories in product data. I noticed that using subcategories as labels delivers worse results ...
  • 161
6 votes
2 answers
101 views

Data transformations in hierarchical classification

I am building a hierarchical text classifier using the Local Classifier Per Parent Node (LCPN) approach with the 'siblings' policy as described in the A survey of hierarchical classification across ...
  • 161
5 votes
2 answers
8k views

Text similarity using RNN

Data set contains records of short text, typically a sentence. The goal is to find duplicated records and similar records. Currently, I have tried R package 'text2vec', the glove word vectors and the ...
5 votes
3 answers
2k views

How can I group texts with similar content together?

I need to find a solution to group a corpus of texts according to document similarity. Premising I have no experience in ML - only a few readings - I'd like to know if calculating the tf-idf on each ...
  • 191
5 votes
1 answer
777 views

How to evaluate the similarity of two columns containing strings?

I am new to text processing and stuck on a problem to identify the similarity of columns. To detail the problem, consider we have two columns with string values: ...
5 votes
0 answers
2k views

What is the minimum number of times a word needs to appear in word2vec training corpus for quality results?

When training a word2vec model with, eg, gensim, you can specify the minimum times a word needs to be seen (with the parameter min_count). The default value for this seems to be 5. Are there any ...
4 votes
1 answer
55 views

encoding of text data in NLP

I'm getting data using web scraping to create a dataset. I have a 'company' column that contains the names of the companies. I would like to encode this column but i don't know how to find the ...
  • 43
4 votes
3 answers
2k views

Bidirectional Encoder Representations from Transformers in R

Can anybody suggest to me, where I can find example code for R language for BERT neural network for text mining tasks. All I can see are python examples, and I need R. https://github.com/google-...
's user avatar
4 votes
2 answers
184 views

Automatic annotation of medical text data

I have a dataset of 30000 whole-genome sequence analysis. For each sample I have a text description that does not follow any fixed format. What I want, is an annotation of each sample indicating the ...
4 votes
1 answer
3k views

Doc2vec to calculate cosine similarity - absolutely inaccurate

I'm trying to modify the Doc2vec tutorial to calculate cosine similarity and take Pandas dataframes instead of .txt documents. I ...
  • 1,268
4 votes
4 answers
229 views

Extract 2 pieces of information from a string - what to use?

First of all, I am a complete newbie in regard to data science and I am not asking for the complete solution but some guidance as to what I should read up to achieve my task (what algorithms, ...
  • 141
3 votes
2 answers
81 views

Multiclass classification of textual data

I have a problem statement in which I have to classify the text data into various classes, but the training data is very less (250-300 data points for 4 classes). I am confused about what approach to ...
  • 31
3 votes
1 answer
122 views

How to classify support call texts?

I have a spreadsheet with thousands of records regarding support requests. Case number, Issue description, etc. Our goal is to classify these records in many categories in order to assign them the ...
  • 719
3 votes
1 answer
173 views

Changing multiple models into 1 model

I am working for a recruitment company on developing machine learning algorithms to automatically classify job applicants as either to be interviewed or not be interveiwed. The data is highly ...
  • 141
3 votes
1 answer
9k views

How does ,the Mutlinomial Bayes's alpha parameter, affects the text classification task?

I would like to know how the alpha parameter, in Multinomial Bayes, affects the text classification task. I know that this parameter is correlated to the algorithm'...
  • 663
3 votes
2 answers
1k views

Word analysis in Python

I have a list of documents which look like this: ...
  • 719
3 votes
1 answer
486 views

Extract date/duration from text

The text and output to be extracted are similar to the following : "Check it every two weeks" - two weeks "Check it on day 1 and day 14" - day 1 and day 14 "day 19 and day ...
3 votes
1 answer
51 views

Classifying one particular class of documents from the rest

I am trying to build a classifier that would classify if a document is a document about sports or not. I have enough samples of sports document to train a classifier on, however I can't imagine how I ...
3 votes
1 answer
2k views

Is there any clear tutorial for how to use AutoEncoders with text as input

I have a pandas dataframe that describes some fields of the register. I have used one hot encoding to encode the feature vectors that are not numbers. Finally my dataset now has 4000 rows * 4 columns. ...
3 votes
1 answer
34 views

NLP text representation techniques that preserve word order in sentence?

I see people are talking mostly about bag-of-words, td-idf and word embeddings. But these are at word levels. BoW and tf-idf fail to represent word orders, and word embeddings are not meant to ...
  • 359
3 votes
1 answer
26 views

Text vectorizer that capture feature offset in the text?

I'm using sklearn Tfifdfvectorizer to extract feature from text towards text classification. I believe the information I need tends to be in the beginning of the document, so I would like to somehow ...
  • 53
3 votes
2 answers
1k views

How to extract and classify data from a column in excel?

I have a column in an Excel sheet that contains a lot of data separated by || delimiters. The data can be classified to some classes like Entity, IFSC codes, ...
3 votes
1 answer
5k views

Choosing the right parameters to train a Tf-Idf vectoriser

I'm very new to the DS world, so please bear with my ignorance. I'm trying to analyse user comments in Spanish. I have a somewhat small dataset (in the 100k's -- is that small?), and when I run the ...
2 votes
2 answers
499 views

Suggestions for guided NLP online courses - Beginner 101

I would like to know from the data science community here for suggestions on nlp courses. I am new to NLP area and would like to take up a course which covers from basic to advanced concepts such as ...
  • 2,337
2 votes
1 answer
1k views

How to impute missing text data?

Lets say I have a dataframe consisting of two text columns. By text, I mean the values in those columns are either sentences/paragraphs. In such a case, how do I handle missing 'NaN' values? If it ...
2 votes
1 answer
5k views

One hot encoding at character level with Keras

I am reading Chollet's book on deep learning at the moment and in the NLP chapter he says: ...
2 votes
1 answer
63 views

How to separate words that are together in a large data set

in twitter data i came across words that are glued together like 'boycottbears' i want them as 'boycott' 'bears' 'man' i tried this but this is slow ...
  • 21
2 votes
1 answer
50 views

What is the best approach for classifying non-English text

What would be the best approach for classifying non-English (Sinhala / Tamil) text? Currently I use Fasttext. Are there any better options? I want to classify user questions into chatbot intents. ...
  • 143
2 votes
2 answers
855 views

Is there any library available for balancing imbalanced text dataset?

I have a text dataset similar to newsgroup dataset, the problem with the dataset is that it is highly imbalanced. So is there any readily built library that will do upsampling or downsampling with a ...
2 votes
2 answers
983 views

Accuracy reduces drastically when using TruncatedSVD with hashingvector

I have around 0.8 million product description with categories. There are around 280 categories. I want to train a model with given dataset so that in future I can predict Category for the given ...
2 votes
1 answer
3k views

What are some function/package in R to find similarity of individual words not in the context of sentences?

what are some function/package in R/python to find similarity of individual words not in the context of sentences? As a novice, I searched and found methods like adist, cosine similarity, word2vec, ...
  • 121
2 votes
3 answers
171 views

Match a two items from two different receipts

I have two different invoices or receipts. One is a Purchase order one is something like a receipt(acknowledgement). Suppose I have ordered(PO) Wine: White Wine Red Wine Rose Wine And I receive ...
2 votes
1 answer
240 views

CNN to many outputs

I have a dataset with 100 columns (categorial one-hot encoded) and 1 column with text data (simple sentences) and i want to build a neural network to arround 380.000 outputs labels. I have no idea ...
2 votes
2 answers
480 views

Discovering string "motifs" in python

I have millions of strings from different sources that tend to exhibit some common patterns. Is there a way to extract these common motifs? For example, in a list (of millions) that includes strings ...
2 votes
1 answer
289 views

Are there any open-source text annotation for multi label classification tools?

I have a large texts in each document and I want to know if there are any open source text annotation tools available online for multiple label annotation. Each sentence takes two labels. If there ...
  • 337
2 votes
1 answer
110 views

Why do probabilities sum to one and how can I set optimal threshold level?

I am working on a text classification use case. The training data has two classes, so the XBBoostClassifier and onevsrest model is classifying the test data into either of the two classes. But my ...
2 votes
2 answers
921 views

How to extract keywords from a list of URLs?

I have a bunch of URLs in a text file like- ...
  • 121
2 votes
1 answer
396 views

Implementing back translation as a data augmentation for text classification

Since back translation English->other language -> English seems like quite a useful data augmentation technique , I wanted to experiment with it. E.g. it occurred to me that languages from very ...
  • 363
2 votes
0 answers
23 views

Classifying short strings of text with additional context

I have a list of short strings each identifying a city. Misspellings are very common. The example below shows some of these short strings, along with the correct city they're supposed to match. ...
  • 165
2 votes
2 answers
613 views

Clustering mixed data types - numeric, categorical, arrays, and text

I have a dataset with 4 types of data columns: ...
  • 121
2 votes
1 answer
236 views

How to implement LSTM using Doc2Vec vectors to get representation?

Hi all. I'm a newbie in ML. I read and found a paper about A Multi-Level Plagiarism Detection System Based on Deep Learning Algorithms and want to implement this model . But I can't find more about ...
2 votes
1 answer
243 views

Needed: Java library to calculate text readability/complexity

In principle the same as this but for Java (and ideally for multiple languages) (e.g. flesch reading ease, smog index, flesch kincaid grade, coleman liau index, automated readability index, dale chall ...
  • 244
2 votes
1 answer
458 views

Why is spam detection a classification problem and not a class modelling problem

Trying to get my feet wet with machine learning on text. The most common dataset I've seen in this space is the sms dataset with classes ham and spam. And the most common and successful approach ...
1 vote
3 answers
7k views

Grouping of similar looking text

I have a data frame which has two columns, "Title" and "Description". The title column has a bunch of titles related to clinical lab tests. Unfortunately, most of the titles are a repeat of the same ...
1 vote
4 answers
392 views

Is it OK to train a binary classifier using all the extremely imbalanced data if the majority class is negative?

I'm training a neural network as a binary classifier for text classification. The data is very imbalanced, where the ratio of TRUE:FALSE is approximately 100:10000 Intuitively, it feels like using ...
1 vote
2 answers
54 views

Grouping company information

I have 3 different datasets with company information, in all of them I have company name, but is not perfect: For example: Dataset A: Company name: Facebook Dataset B: Company name: Facebook, Inc ...
  • 719