16 votes
Accepted

Keyword/phrase extraction from Text using Deep Learning libraries

The Google Research Blog should be helpful in the context of TensorFlow. In the above article, there is a reference to the Annotated English Gigaword dataset which is routinely used for text ...
Society of Data Scientists's user avatar
16 votes

Keyword/phrase extraction from Text using Deep Learning libraries

This is an open area of research and it certainly depends on the way you frame the problem. If you're talking about multi-document summarization then the problem is slightly different than if you were ...
franciscojavierarceo's user avatar
13 votes
Accepted

How to determine if character sequence is English word or noise

During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following. ngrams Take a random sample of ...
Brandon Loudermilk's user avatar
12 votes
Accepted

What is the difference between NLP and text mining?

I agree with Sean's answer. NLP and text mining are usually used for different goals. Also, there is indeed an overlap and both definitions are vogue. Other than the difference in goal, there is a ...
DaL's user avatar
  • 2,623
12 votes
Accepted

Doc2Vec - How to label the paragraphs (gensim)

Both are possible. You can give every document a unique ID (such as a sequential serial number) as a doctag, or a shared string doctag representing something else about it, or both at the same time. ...
gojomo's user avatar
  • 236
12 votes

What is Hellinger Distance and when to use it?

Hellinger distance is a metric to measure the difference between two probability distributions. It is the probabilistic analog of Euclidean distance. Given two probability distributions, $P$ and $Q$, ...
Brian Spiering's user avatar
12 votes

Clustering with cosine similarity

First, every clustering algorithm is using some sort of distance metric. Which is actually important, because every metric has its own properties and is suitable for different kind of problems. You ...
HonzaB's user avatar
  • 1,669
12 votes
Accepted

Resume Parsing - extracting skills from resume using Machine Learning

I'm not sure Topic Modelling will help you here, as it tries to extract abstract topics from text. I'm afraid resumes might be too 'dry' for it to work nicely. Here are a few sources I found that ...
xhattam's user avatar
  • 136
12 votes

How to automatically classify a sentence or text based on its context?

Me: Please give 2 semantic tags for the sentence "The area of a circle is pi time the radius squared" ChatGPT: 1. Mathematics. 2. Geometry I'm not sure it's a robust and scalable solution ...
Sergey Skripko's user avatar
11 votes

Doc2Vec - How to label the paragraphs (gensim)

doc2vec model gets its algorithm from word2vec. In word2vec there is no need to label the ...
chmodsss's user avatar
  • 1,954
11 votes
Accepted

What is the difference between a hashing vectorizer and a tfidf vectorizer

The main difference is that HashingVectorizer applies a hashing function to term frequency counts in each document, where ...
redhqs's user avatar
  • 1,658
10 votes

How to do postal addresses fuzzy matching?

As you are using R you might want to look into the stringdist package and the Jaro-Winkler distance metric that can be used in the calculations. This was developed at the U.S. Census Bureau for ...
phiver's user avatar
  • 718
9 votes

applying word2vec on small text files

Word2Vec isn't a good choice for a dataset of such size. From researches I have seen, it will unleash its power if you feed at least couple of million of words, 3k tweets wouldn't be enough for a ...
chewpakabra's user avatar
9 votes

What is the difference between NLP and text mining?

I have had this doubt since a long time. So, this post here helped me figure the differences between the two. So, this is the difference between text mining and NLP: Text Mining deals with the ...
Dawny33's user avatar
  • 8,246
9 votes

How to deal with spelling errors NLP

Other options would be to... Compare similar text sequences Compare similar string sequences Use fuzzy matching Fuzzy Matching: ...
Peter's user avatar
  • 7,297
9 votes
Accepted

How to automatically classify a sentence or text based on its context?

To my knowledge, there is no such library or pre-trained model. Imho there is an important issue in the task as defined in the question, more exactly in the example: these tags seem natural for a ...
Erwan's user avatar
  • 24.8k
8 votes

Document classification using convolutional neural network

You could reduce the length of your input data by representing your documents as series of sentence vectors instead of a longer series of word vectors. Doc2vec is one way to do this (each sentence ...
Andrew's user avatar
  • 256
8 votes
Accepted

How to determine the complexity of an English sentence?

Yes. There are various metrics, such as the fogg index. Textacy in python has a nice list and implementations. ...
GrimSqueaker's user avatar
8 votes

Date Extraction in Python

Stanford CoreNLP has a very good implementation of NER for date/time. https://nlp.stanford.edu/software/sutime.html (demo: http://nlp.stanford.edu:8080/sutime/process) Though this is written in ...
Shamit Verma's user avatar
  • 2,239
7 votes

How to give name to topics created using LDA?

I can suggest several papers on this topic: Automatic Labelling of Topic Models Automatic Labeling Hierarchical Topics Representing Topics Labels for Exploring Digital Libraries You can find more by ...
Emre's user avatar
  • 10.5k
7 votes
Accepted

How evaluate text clustering?

Check out this paper. It also addresses question of how many clusters to use. The R package mclust has a routine which will try different cluster models/number of clusters and plot the Bayesian ...
Pete's user avatar
  • 809
7 votes
Accepted

Text extraction from documents using NLP or Deep Learning

Jurafsky and Martin's NLP textbook has a chapter about information extraction that should be a good starting point. For example, if you want to extract company names it will tell you how to do that. ...
polm23's user avatar
  • 343
7 votes
Accepted

How to measure the similarity between two text documents?

In general,there are two ways for finding document-document similarity TF-IDF approach Make a text corpus containing all words of documents . You have to use tokenisation and stop word removal . ...
Pankaj Kumar's user avatar
6 votes

What is the difference between a hashing vectorizer and a tfidf vectorizer

The HashingVectorizer has a parameter n_features which is 1048576 by default. When hashing, ...
Nathan's user avatar
  • 160
6 votes
Accepted

Public dataset for news articles with their associated categories

Here is a massive dataset of news with categories which I created for exactly such a reason. Includes all the headlines published by Times of India from 2001-2019 with categories. Contains ~3 million ...
Rohit's user avatar
  • 176
6 votes
Accepted

Handling data imbalance and class number for classification

People talk a lot about data imbalance, but in general I think you don't need to worry about it unless your data is really imbalanced (like <1% of one label). 50/200 is fine. If you build a ...
tom's user avatar
  • 2,238
5 votes
Accepted

Appropriate algorithm for string (not document) classification?

You might find it useful to treat n-grams of characters as your feature space. Then you could represent a string as a bag of substrings. With N = 4 or greater, you ...
jamesmf's user avatar
  • 3,077
5 votes

Choosing class labels from annotated data

That's one valid way of approaching the problem. In your final solution, though, it will be helpful to quantify the overall inter-rater agreement. For example, Cohen's kappa is a commonly-used metric: ...
Kyle.'s user avatar
  • 1,483
5 votes

Text-Classification-Problem: Is Word2Vec/NN the best approach?

1) Max-Entropy(Logistic Regression) on TFIDF vectors is a good starting point for many NLP classification task. 2) Word2vec is definitely something worth trying and comparing to model 1. I would ...
rushimg's user avatar
  • 51
5 votes

Unsupervised feature learning for NER

In this 2014 paper (GitHub), the authors compared multiple strategies of incorporating word embeddings in a CRF-based NER system, including dense embedding, binerized embedding, cluster embedding, and ...
user2404894's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible