Skip to main content
16 votes

Keyword/phrase extraction from Text using Deep Learning libraries

This is an open area of research and it certainly depends on the way you frame the problem. If you're talking about multi-document summarization then the problem is slightly different than if you were ...
franciscojavierarceo's user avatar
16 votes
Accepted

Keyword/phrase extraction from Text using Deep Learning libraries

The Google Research Blog should be helpful in the context of TensorFlow. In the above article, there is a reference to the Annotated English Gigaword dataset which is routinely used for text ...
Society of Data Scientists's user avatar
13 votes
Accepted

How to determine if character sequence is English word or noise

During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following. ngrams Take a random sample of ...
Brandon Loudermilk's user avatar
13 votes
Accepted

Resume Parsing - extracting skills from resume using Machine Learning

I'm not sure Topic Modelling will help you here, as it tries to extract abstract topics from text. I'm afraid resumes might be too 'dry' for it to work nicely. Here are a few sources I found that ...
xhattam's user avatar
  • 146
12 votes

What is Hellinger Distance and when to use it?

Hellinger distance is a metric to measure the difference between two probability distributions. It is the probabilistic analog of Euclidean distance. Given two probability distributions, $P$ and $Q$, ...
Brian Spiering's user avatar
12 votes

Clustering with cosine similarity

First, every clustering algorithm is using some sort of distance metric. Which is actually important, because every metric has its own properties and is suitable for different kind of problems. You ...
HonzaB's user avatar
  • 1,669
12 votes

How to automatically classify a sentence or text based on its context?

Me: Please give 2 semantic tags for the sentence "The area of a circle is pi time the radius squared" ChatGPT: 1. Mathematics. 2. Geometry I'm not sure it's a robust and scalable solution ...
Sergey Skripko's user avatar
11 votes
Accepted

What is the difference between a hashing vectorizer and a tfidf vectorizer

The main difference is that HashingVectorizer applies a hashing function to term frequency counts in each document, where ...
redhqs's user avatar
  • 1,678
10 votes

How to deal with spelling errors NLP

Other options would be to... Compare similar text sequences Compare similar string sequences Use fuzzy matching Fuzzy Matching: ...
Peter's user avatar
  • 7,526
9 votes
Accepted

How to automatically classify a sentence or text based on its context?

To my knowledge, there is no such library or pre-trained model. Imho there is an important issue in the task as defined in the question, more exactly in the example: these tags seem natural for a ...
Erwan's user avatar
  • 25.5k
8 votes

Document classification using convolutional neural network

You could reduce the length of your input data by representing your documents as series of sentence vectors instead of a longer series of word vectors. Doc2vec is one way to do this (each sentence ...
Andrew's user avatar
  • 256
8 votes
Accepted

How to determine the complexity of an English sentence?

Yes. There are various metrics, such as the fogg index. Textacy in python has a nice list and implementations. ...
GrimSqueaker's user avatar
8 votes

Date Extraction in Python

Stanford CoreNLP has a very good implementation of NER for date/time. https://nlp.stanford.edu/software/sutime.html (demo: http://nlp.stanford.edu:8080/sutime/process) Though this is written in ...
Shamit Verma's user avatar
  • 2,259
7 votes
Accepted

How evaluate text clustering?

Check out this paper. It also addresses question of how many clusters to use. The R package mclust has a routine which will try different cluster models/number of clusters and plot the Bayesian ...
Pete's user avatar
  • 819
7 votes
Accepted

Text extraction from documents using NLP or Deep Learning

Jurafsky and Martin's NLP textbook has a chapter about information extraction that should be a good starting point. For example, if you want to extract company names it will tell you how to do that. ...
polm23's user avatar
  • 343
7 votes
Accepted

How to measure the similarity between two text documents?

In general,there are two ways for finding document-document similarity TF-IDF approach Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . ...
Pankaj Kumar's user avatar
6 votes

What is the difference between a hashing vectorizer and a tfidf vectorizer

The HashingVectorizer has a parameter n_features which is 1048576 by default. When hashing, ...
Nathan's user avatar
  • 160
6 votes
Accepted

Public dataset for news articles with their associated categories

Here is a massive dataset of news with categories which I created for exactly such a reason. Includes all the headlines published by Times of India from 2001-2019 with categories. Contains ~3 million ...
Rohit's user avatar
  • 176
6 votes
Accepted

Handling data imbalance and class number for classification

People talk a lot about data imbalance, but in general I think you don't need to worry about it unless your data is really imbalanced (like <1% of one label). 50/200 is fine. If you build a ...
tom's user avatar
  • 2,248
5 votes

Unsupervised feature learning for NER

In this 2014 paper (GitHub), the authors compared multiple strategies of incorporating word embeddings in a CRF-based NER system, including dense embedding, binerized embedding, cluster embedding, and ...
user2404894's user avatar
5 votes
Accepted

What is a good explanation of Non Negative Matrix Factorization?

Non-Negative Matrix Factorization (NMF) is described well in the paper by Lee and Seung, 1999. Simply Put NMF takes as an input a term-document matrix and generates a set of topics that represent ...
Society of Data Scientists's user avatar
5 votes

Keyword/phrase extraction from Text using Deep Learning libraries

Sounds like this is more extractive summarization if you are looking for key words. Here are a few papers which probably have implementations: Neural Summarization by Extracting Sentences and Words ...
Pavel Savine's user avatar
5 votes
Accepted

Information extraction with reinforcement learning, feasible?

You ideally want to use Reinforcement Learning in situations where there is delayed feedback and stochastic transitions in the environment. Although you could potentially apply RL, in your case, you ...
Karthik's user avatar
  • 66
5 votes

Clustering with cosine similarity

I'd use sklearn's Hierarchical clustering ...
Uri Goren's user avatar
  • 438
5 votes

What algorithm can help me discover synonyms?

word2vec is probably the way to go. It maps words to a point in n-dimensional space. You can use Euclidean (or whatever distance) to find the nearest points to a given word. If training went well, the ...
Kyle's user avatar
  • 51
5 votes

Public dataset for news articles with their associated categories

This dataset is included with scikit-learn, a popular ML library for Python. It is postings to Usenet and categorized by the group. The group titles are not exactly "categories" like you ...
CalZ's user avatar
  • 1,663
5 votes
Accepted

How does keras calculate accuracy for multi label classification?

I have answered a similar question here. Your problem is that Accuracy is not the right metric for multi-label tasks. Try something different like AUC, precision, recall, accuracy@k, precision@recall. ...
Gianmario Spacagna's user avatar
5 votes
Accepted

How to use different classes of words in CountVectorizer()

First of all your question is about stemming words as mentioned in the other answer which can be found in any Python NLP library such as Spacy or NLTK. The other point to mention here is that despite ...
Kasra Manshaei's user avatar
5 votes
Accepted

which deep learning text classifier is good for health data

Yes, you should split the paragraph to sentences and give those sentences to the model. Your deep structure should be like this: In the first layer, you must put a word embedding layer to represent a ...
pythinker's user avatar
  • 1,267
5 votes

How to deal with spelling errors NLP

2 approaches to correct misspellings: Make your own dictionary of corrections, for example: ...
Noah Weber's user avatar
  • 5,699

Only top scored, non community-wiki answers of a minimum length are eligible