40
votes
Accepted
Sentence similarity prediction
Your problem can be solved with Word2vec as well as Doc2vec. Doc2vec would give better results because it takes sentences into account while training the model.
Doc2vec solution
You can train your ...
- 706
16
votes
How do you apply SMOTE on text classification?
Actually NLP is one of the most common areas in which resampling of data is needed as there are many text classification tasks dealing with imbalanced problem (think of spam filtering, insulting ...
- 6,712
10
votes
Sentence similarity prediction
Word Mover’s Distance (WMD) is an algorithm for finding the distance between sentences. WMD is based on word embeddings (e.g., word2vec) which encode the semantic meaning of words into dense vectors.
...
- 22.3k
7
votes
Accepted
How to impute missing text data?
First most of the time there's no "missing text", there's an empty string (0 sentences, 0 words) and this is a valid text value. The distinction is important, because the former usually ...
- 25.9k
5
votes
Sentence similarity prediction
You can try an easy solution using sklearn and it's going to work fine.
Use tfidfvectorizer to get a vector representation of each text
Fit the vectorizer with your data, removing stop-words.
...
- 760
5
votes
Accepted
One hot encoding at character level with Keras
I think that you are looking for the keras Tokenizer with the char_level=True flag:
...
- 1,123
5
votes
Accepted
Suggestions for guided NLP online courses - Beginner 101
I would recommend two course which focus on code first approach and which will help you understand concepts by getting your hands dirty. Both of these courses contains code and video resources.
Fast....
- 1,521
5
votes
Suggestions for guided NLP online courses - Beginner 101
I recommend Manning's course, the course is available for free on youtube. However it doesn't really start from zero, it's quite advanced imho.
- 25.9k
4
votes
Which type auto encoder gives best results for text
A working example of a Variational Autoencoder for Text Generation in Keras can be found here.
Cross-entropy loss, aka log loss, measures the performance of a model whose output is a probability ...
- 22.3k
4
votes
Accepted
nltk's stopwords returns "TypeError: argument of type 'LazyCorpusLoader' is not iterable"
There are a couple of items that could be improved in your code:
nltk.corpus.stopwords is a nltk.corpus.util.LazyCorpusLoader. ...
- 22.3k
4
votes
How to separate words that are together in a large data set
This commonly called a "word break" problem. There are a variety of approaches, the most common use dynamic programming or tries. You can recursively try candidates and keep the candidates if they can ...
- 22.3k
4
votes
Accepted
Why is dictionary-based approach a heuristic method?
My opinion is that besides intellectual gymnastics, the difference between the two doesn't matter much in practice. To me, it's mostly semantics as I would and have used the two interchangeably.
The ...
- 6,106
3
votes
How do you apply SMOTE on text classification?
If you want to add more text/sentences traning data, you can use pre-trained word embeddings. Pretrained models like provides word vector representation of each and every dictionary word. It also ...
3
votes
Accepted
3
votes
What are some function/package in R to find similarity of individual words not in the context of sentences?
If your intent is to find compare similarity in meaning, word2vec is the only appropriate choice. adist measures the edit distance between two words, and cosine similarity compares the similarity of ...
- 391
3
votes
Text similarity using RNN
Doc2Vec, Mikolov's paper will solve your problem. Here is the paper. You can find a gensim implementationhere. While using RNN, using GLOVE or Googl Word2Vec will be always useful even if your ...
- 1,858
3
votes
Accepted
How to use correlation matrix when the dataset contains multiple columns with text data?
The problem is that the correlation matrix has to be done with numerical values. So what you have to do is to transform the texts into numerical vectors. There are several ways of doing this, there ...
- 4,243
3
votes
Bidirectional Encoder Representations from Transformers in R
You might be interested in the open-source R package RBERT:
https://github.com/jonathanbratt/RBERT
It's a work in progress, but the goal is to be able to use BERT directly in R.
- 31
3
votes
Accepted
What is the best approach for classifying non-English text
As far as I know, the best way would be to use pretrained embedder. Embedder encodes your text into language-agnostic latent space. You input your text and you get fixed-length numerical vector as an ...
- 844
3
votes
Accepted
encoding of text data in NLP
This kind of problem is called record linkage (or sometimes entity matching or other variants). The task consists in finding among a list of strings representing entities (persons or organizations) ...
- 25.9k
3
votes
How to evaluate the similarity of two columns containing strings?
You could use similarity metrics for strings. There are a number of "off the shelf" packages to compare string similarity, such as stringdist for ...
- 7,776
2
votes
Accepted
Text Mining of Research Paper Abstracts
In order to train some supervised learning algorithm to identify 'Problem' and 'Solution', you need to somehow generate some data that has labels of these things, which may be your best bet. So you ...
- 170
2
votes
Sentence similarity prediction
There is some recent work based on Variational Auto-Encoder in RNN models.Generating Sentences from a Continuous Space, with pytorch implementations: github code.
they managed to compress the semantic,...
- 904
2
votes
Accepted
Text processing
Since you are going to use TF-IDF representations, you already have a feature matrix. To calculate cosine similairty between all vectors, you can use:
...
- 1,858
2
votes
Text similarity using RNN
Unless you have a lot of data, I have my doubts whether training RNNs for similarity will give you significant improvements.
As a baseline, I would go the traditional way first and engineer some ...
- 236
2
votes
Discovering string "motifs" in python
Assuming that letters are indicative of "motifs" and numbers are considered as digits and not exact numbers, this is what I would do:
First - transform numbers into a digit placeholder (...
- 438
2
votes
Accepted
How to add incorporate meta data into text classification?
Some models cannot really handle this, while others lend themselves for it easily. I'll explain two approaches that you could use:
Naive bayes
With Naive Bayes you can use other categorical values ...
- 9,398
2
votes
Accepted
Classify text labels in to a similar category
One way to do it could be with fuzzy string search. Levenshtein distance algorithm is what you may use for it.
... the Levenshtein distance is a string metric for measuring the difference between ...
- 168
2
votes
Grouping of similar looking text
Pandas can directly do that string compare, then use the compare result to lookup appropriate rows so that they can be set. This can be done with a single expression as:
Code:
...
- 1,811
2
votes
Accepted
How does ,the Mutlinomial Bayes's alpha parameter, affects the text classification task?
Lets assume you are building a text classifier with a training set of 5 sentences. For this example, lets say you are trying to classify tweets (which are usually a sentence long) to whether it was a ...
- 146
Only top scored, non community-wiki answers of a minimum length are eligible