Skip to main content
Share Your Experience: Take the 2024 Developer Survey
40 votes
Accepted

Sentence similarity prediction

Your problem can be solved with Word2vec as well as Doc2vec. Doc2vec would give better results because it takes sentences into account while training the model. Doc2vec solution You can train your ...
Harman's user avatar
  • 706
16 votes

How do you apply SMOTE on text classification?

Actually NLP is one of the most common areas in which resampling of data is needed as there are many text classification tasks dealing with imbalanced problem (think of spam filtering, insulting ...
Kasra Manshaei's user avatar
10 votes

Sentence similarity prediction

Word Mover’s Distance (WMD) is an algorithm for finding the distance between sentences. WMD is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors. ...
Brian Spiering's user avatar
7 votes
Accepted

How to impute missing text data?

First most of the time there's no "missing text", there's an empty string (0 sentences, 0 words) and this is a valid text value. The distinction is important, because the former usually ...
Erwan's user avatar
  • 25.5k
5 votes

Sentence similarity prediction

You can try an easy solution using sklearn and it's going to work fine. Use tfidfvectorizer to get a vector representation of each text Fit the vectorizer with your data, removing stop-words. ...
Federico Caccia's user avatar
5 votes
Accepted

One hot encoding at character level with Keras

I think that you are looking for the keras Tokenizer with the char_level=True flag: ...
Adrien D's user avatar
  • 1,113
5 votes
Accepted

Suggestions for guided NLP online courses - Beginner 101

I would recommend two course which focus on code first approach and which will help you understand concepts by getting your hands dirty. Both of these courses contains code and video resources. Fast....
prashant0598's user avatar
  • 1,511
5 votes

Suggestions for guided NLP online courses - Beginner 101

I recommend Manning's course, the course is available for free on youtube. However it doesn't really start from zero, it's quite advanced imho.
Erwan's user avatar
  • 25.5k
4 votes

How can I group texts with similar content together?

Well, after further googling I found the solution: MinHash or SimHash will do the job and I also found a tool implementing MinHash written in Scala on GitHub right at this link
Max's user avatar
  • 191
4 votes

Which type auto encoder gives best results for text

A working example of a Variational Autoencoder for Text Generation in Keras can be found here. Cross-entropy loss, aka log loss, measures the performance of a model whose output is a probability ...
Brian Spiering's user avatar
4 votes
Accepted

nltk's stopwords returns "TypeError: argument of type 'LazyCorpusLoader' is not iterable"

There are a couple of items that could be improved in your code: nltk.corpus.stopwords is a nltk.corpus.util.LazyCorpusLoader. ...
Brian Spiering's user avatar
4 votes
Accepted

Why is dictionary-based approach a heuristic method?

My opinion is that besides intellectual gymnastics, the difference between the two doesn't matter much in practice. To me, it's mostly semantics as I would and have used the two interchangeably. The ...
Valentin Calomme's user avatar
3 votes

What are some function/package in R to find similarity of individual words not in the context of sentences?

If your intent is to find compare similarity in meaning, word2vec is the only appropriate choice. adist measures the edit distance between two words, and cosine similarity compares the similarity of ...
liangjy's user avatar
  • 391
3 votes

Text similarities

Since your question is super-huge and no one is going to do this job for you, here's a starting point for you to at least let you see how the various string comparison algorithms work: ...
hrbrmstr's user avatar
  • 351
3 votes

Text similarity using RNN

Doc2Vec, Mikolov's paper will solve your problem. Here is the paper. You can find a gensim implementationhere. While using RNN, using GLOVE or Googl Word2Vec will be always useful even if your ...
Himanshu Rai's user avatar
  • 1,838
3 votes
Accepted

How to classify support call texts?

Rather than use an external dictionary of keywords that are indicative of target class, you may want to take your raw data (or rather a random subset of it), then hand-label your instances (assign ...
Brandon Loudermilk's user avatar
3 votes
Accepted

Grouping of similar looking text

One possible solution is the following: ...
Dani Mesejo's user avatar
  • 2,226
3 votes

How do you apply SMOTE on text classification?

If you want to add more text/sentences traning data, you can use pre-trained word embeddings. Pretrained models like provides word vector representation of each and every dictionary word. It also ...
Yashodhan Pawar's user avatar
3 votes
Accepted

How to use correlation matrix when the dataset contains multiple columns with text data?

The problem is that the correlation matrix has to be done with numerical values. So what you have to do is to transform the texts into numerical vectors. There are several ways of doing this, there ...
Simon Larsson's user avatar
3 votes

Bidirectional Encoder Representations from Transformers in R

You might be interested in the open-source R package RBERT: https://github.com/jonathanbratt/RBERT It's a work in progress, but the goal is to be able to use BERT directly in R.
jbratt's user avatar
  • 31
3 votes
Accepted

What is the best approach for classifying non-English text

As far as I know, the best way would be to use pretrained embedder. Embedder encodes your text into language-agnostic latent space. You input your text and you get fixed-length numerical vector as an ...
Piotr Rarus's user avatar
3 votes

How to separate words that are together in a large data set

This commonly called a "word break" problem. There are a variety of approaches, the most common use dynamic programming or tries. You can recursively try candidates and keep the candidates if they can ...
Brian Spiering's user avatar
3 votes
Accepted

encoding of text data in NLP

This kind of problem is called record linkage (or sometimes entity matching or other variants). The task consists in finding among a list of strings representing entities (persons or organizations) ...
Erwan's user avatar
  • 25.5k
3 votes

How to evaluate the similarity of two columns containing strings?

You could use similarity metrics for strings. There are a number of "off the shelf" packages to compare string similarity, such as stringdist for ...
Peter's user avatar
  • 7,526
2 votes

Automatic annotation of medical text data

You could use linear regression on the genome sequence to predict the occurrence of words in the description. More specifically: Use dummy variables to encode the genome sequence. Use stemming to ...
Pieter's user avatar
  • 961
2 votes
Accepted

Text Mining of Research Paper Abstracts

In order to train some supervised learning algorithm to identify 'Problem' and 'Solution', you need to somehow generate some data that has labels of these things, which may be your best bet. So you ...
CHP's user avatar
  • 170
2 votes
Accepted

Text processing

Since you are going to use TF-IDF representations, you already have a feature matrix. To calculate cosine similairty between all vectors, you can use: ...
Himanshu Rai's user avatar
  • 1,838
2 votes

Text similarity using RNN

Unless you have a lot of data, I have my doubts whether training RNNs for similarity will give you significant improvements. As a baseline, I would go the traditional way first and engineer some ...
Alexander Bauer's user avatar
2 votes

Sentence similarity prediction

There is some recent work based on Variational Auto-Encoder in RNN models.Generating Sentences from a Continuous Space, with pytorch implementations: github code. they managed to compress the semantic,...
Fadi Bakoura's user avatar
2 votes

Discovering string "motifs" in python

Assuming that letters are indicative of "motifs" and numbers are considered as digits and not exact numbers, this is what I would do: First - transform numbers into a digit placeholder (...
Uri Goren's user avatar
  • 438

Only top scored, non community-wiki answers of a minimum length are eligible