52 votes
Accepted

What are some standard ways of computing the distance between documents?

There's a number of different ways of going about this depending on exactly how much semantic information you want to retain and how easy your documents are to tokenize (html documents would probably ...
user avatar
  • 4,179
34 votes
Accepted

What is difference between text classification and topic models?

Text Classification I give you a bunch of documents, each of which has a label attached. I ask you to learn why you think the contents of the documents have been given these labels based on their ...
user avatar
  • 6,415
32 votes

General approach to extract key text from sentence (nlp)

Shallow Natural Language Processing technique can be used to extract concepts from sentence. ------------------------------------------- Shallow NLP technique steps: Convert the sentence to lowercase ...
user avatar
25 votes
Accepted

Extract most informative parts of text from documents

What you're describing is often achieved using a simple combination of TF-IDF and extractive summarization. In a nutshell, TF-IDF tells you the relative importance of each word in each document, in ...
user avatar
16 votes

How to annotate text documents with meta-data?

Personally I would advocate using something that is both not-specific to the NLP field, and something that is sufficiently general that it can still be used as a tool even when you've started moving ...
user avatar
  • 4,179
16 votes
Accepted

Keyword/phrase extraction from Text using Deep Learning libraries

The Google Research Blog should be helpful in the context of TensorFlow. In the above article, there is a reference to the Annotated English Gigaword dataset which is routinely used for text ...
user avatar
16 votes

Keyword/phrase extraction from Text using Deep Learning libraries

This is an open area of research and it certainly depends on the way you frame the problem. If you're talking about multi-document summarization then the problem is slightly different than if you were ...
user avatar
15 votes
Accepted

How to grow a list of related words based on initial keywords?

The word2vec algorithm may be a good way to retrieve more elements for a list of similar words. It is an unsupervised "deep learning" algorithm that has previously been demonstrated with Wikipedia-...
user avatar
  • 476
14 votes
Accepted

Suggest text classifier training datasets

Some standard datasets for text classification are the 20-News group, Reuters (with 8 and 52 classes) and WebKb. You can find all of them here.
user avatar
  • 1,516
14 votes

What algorithms should I use to perform job classification based on resume data?

Check out this link. Here, they will take you through loading unstructured text to creating a wordcloud. You can adapt this strategy and instead of creating a wordcloud, you can create a frequency ...
user avatar
  • 485
14 votes

Unstructured text classification

Let's work it out from the ground up. Classification (also known as categorization) is an example of supervised learning. In supervised learning you have: model - something that approximates ...
user avatar
  • 2,771
13 votes
Accepted

What are the main types of NLP annotators?

Here are the basic Natural Language Processing capabilities (or annotators) that are usually necessary to extract language units from textual data for sake of search and other applications: Sentence ...
user avatar
  • 301
13 votes

What is difference between text classification and topic models?

But I don't know what is difference between text classification and topic models in documents Text Classification is a form of supervised learning, hence the set ...
user avatar
13 votes
Accepted

Algorithms for text clustering

Check the Stanford NLP Group's open source software, in particular, Stanford Classifier. The software is written in Java, which will likely delight you, but also ...
user avatar
13 votes
Accepted

How to determine if character sequence is English word or noise

During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following. ngrams Take a random sample of ...
user avatar
12 votes
Accepted

What is the difference between NLP and text mining?

I agree with Sean's answer. NLP and text mining are usually used for different goals. Also, there is indeed an overlap and both definitions are vogue. Other than the difference in goal, there is a ...
user avatar
  • 2,583
12 votes
Accepted

Doc2Vec - How to label the paragraphs (gensim)

Both are possible. You can give every document a unique ID (such as a sequential serial number) as a doctag, or a shared string doctag representing something else about it, or both at the same time. ...
user avatar
  • 236
12 votes
Accepted

Resume Parsing - extracting skills from resume using Machine Learning

I'm not sure Topic Modelling will help you here, as it tries to extract abstract topics from text. I'm afraid resumes might be too 'dry' for it to work nicely. Here are a few sources I found that ...
user avatar
  • 136
11 votes

What algorithms should I use to perform job classification based on resume data?

Just extract keywords and train a classifier on them. That's all, really. Most of the text in CVs is not actually related to skills. E.g. consider sentence "I'm experienced and highly efficient in ...
user avatar
  • 2,771
11 votes

Doc2Vec - How to label the paragraphs (gensim)

doc2vec model gets its algorithm from word2vec. In word2vec there is no need to label the ...
user avatar
  • 1,924
11 votes

What is Hellinger Distance and when to use it?

Hellinger distance is a metric to measure the difference between two probability distributions. It is the probabilistic analog of Euclidean distance. Given two probability distributions, $P$ and $Q$, ...
user avatar
11 votes

Clustering with cosine similarity

First, every clustering algorithm is using some sort of distance metric. Which is actually important, because every metric has its own properties and is suitable for different kind of problems. You ...
user avatar
  • 1,641
10 votes

How to do postal addresses fuzzy matching?

As you are using R you might want to look into the stringdist package and the Jaro-Winkler distance metric that can be used in the calculations. This was developed at the U.S. Census Bureau for ...
user avatar
  • 720
10 votes
Accepted

What is the difference between a hashing vectorizer and a tfidf vectorizer

The main difference is that HashingVectorizer applies a hashing function to term frequency counts in each document, where ...
user avatar
  • 1,558
9 votes
Accepted

R error using package tm (text-mining)

You have to tell Corpus what kind of source you are using. Try: Corpus(VectorSource(d1$Yes))
user avatar
  • 485
9 votes

Preference Matching Algorithm

My first suggestion would be to somehow map the non-quantifiable attributes to quantities with the help of suitable mapping functions. Otherwise, simply leave them out. Secondly, I don't think that ...
user avatar
  • 1,516
9 votes

applying word2vec on small text files

Word2Vec isn't a good choice for a dataset of such size. From researches I have seen, it will unleash its power if you feed at least couple of million of words, 3k tweets wouldn't be enough for a ...
user avatar
9 votes

What is the difference between NLP and text mining?

I have had this doubt since a long time. So, this post here helped me figure the differences between the two. So, this is the difference between text mining and NLP: Text Mining deals with the ...
user avatar
  • 8,016
8 votes
Accepted

Difference between tf-idf and tf with Random Forests

Decision trees (and hence Random Forests) are insensitive to monotone transformations of input features. Since multiplying by the same factor is a monotone transformation, I'd assume that for Random ...
user avatar
8 votes

Which classification algorithms to try for classifying text data into 300 categories

In general, a decent starting point for problems like these is Naive Bayes (NB) classification using a simple bag of words model. Here are some slides describing NB as applied to natural language ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible