52
votes
Accepted
What are some standard ways of computing the distance between documents?
There's a number of different ways of going about this depending on exactly how much semantic information you want to retain and how easy your documents are to tokenize (html documents would probably ...
- 4,179
34
votes
Accepted
What is difference between text classification and topic models?
Text Classification
I give you a bunch of documents, each of which has a label attached. I ask you to learn why you think the contents of the documents have been given these labels based on their ...
- 6,415
32
votes
General approach to extract key text from sentence (nlp)
Shallow Natural Language Processing technique can be used to extract concepts from sentence.
-------------------------------------------
Shallow NLP technique steps:
Convert the sentence to lowercase
...
25
votes
Accepted
Extract most informative parts of text from documents
What you're describing is often achieved using a simple combination of TF-IDF and extractive summarization.
In a nutshell, TF-IDF tells you the relative importance of each word in each document, in ...
- 1,501
16
votes
How to annotate text documents with meta-data?
Personally I would advocate using something that is both not-specific to the NLP field, and something that is sufficiently general that it can still be used as a tool even when you've started moving ...
- 4,179
16
votes
Accepted
Keyword/phrase extraction from Text using Deep Learning libraries
The Google Research Blog should be helpful in the context of TensorFlow.
In the above article, there is a reference to the Annotated English Gigaword dataset which is routinely used for text ...
16
votes
Keyword/phrase extraction from Text using Deep Learning libraries
This is an open area of research and it certainly depends on the way you frame the problem. If you're talking about multi-document summarization then the problem is slightly different than if you were ...
15
votes
Accepted
How to grow a list of related words based on initial keywords?
The word2vec algorithm may be a good way to retrieve more elements for a list of similar words. It is an unsupervised "deep learning" algorithm that has previously been demonstrated with Wikipedia-...
- 476
14
votes
Accepted
Suggest text classifier training datasets
Some standard datasets for text classification are the 20-News group, Reuters (with 8 and 52 classes) and WebKb. You can find all of them here.
- 1,516
14
votes
What algorithms should I use to perform job classification based on resume data?
Check out this link.
Here, they will take you through loading unstructured text to creating a wordcloud. You can adapt this strategy and instead of creating a wordcloud, you can create a frequency ...
- 485
14
votes
Unstructured text classification
Let's work it out from the ground up. Classification (also known as categorization) is an example of supervised learning. In supervised learning you have:
model - something that approximates ...
- 2,771
13
votes
Accepted
What are the main types of NLP annotators?
Here are the basic Natural Language Processing capabilities (or annotators) that are usually necessary to extract language units from textual data for sake of search and other applications:
Sentence ...
- 301
13
votes
What is difference between text classification and topic models?
But I don't know what is difference between text classification and topic models in documents
Text Classification is a form of supervised learning, hence the set ...
- 1,501
13
votes
Accepted
Algorithms for text clustering
Check the Stanford NLP Group's open source software, in particular, Stanford Classifier. The software is written in Java, which will likely delight you, but also ...
- 6,508
13
votes
Accepted
How to determine if character sequence is English word or noise
During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following.
ngrams
Take a random sample of ...
- 1,206
12
votes
Accepted
What is the difference between NLP and text mining?
I agree with Sean's answer.
NLP and text mining are usually used for different goals.
Also, there is indeed an overlap and both definitions are vogue.
Other than the difference in goal, there is a ...
- 2,583
12
votes
Accepted
Doc2Vec - How to label the paragraphs (gensim)
Both are possible. You can give every document a unique ID (such as a sequential serial number) as a doctag, or a shared string doctag representing something else about it, or both at the same time.
...
- 236
12
votes
Accepted
Resume Parsing - extracting skills from resume using Machine Learning
I'm not sure Topic Modelling will help you here, as it tries to extract abstract topics from text. I'm afraid resumes might be too 'dry' for it to work nicely.
Here are a few sources I found that ...
- 136
11
votes
What algorithms should I use to perform job classification based on resume data?
Just extract keywords and train a classifier on them. That's all, really.
Most of the text in CVs is not actually related to skills. E.g. consider sentence "I'm experienced and highly efficient in ...
- 2,771
11
votes
Doc2Vec - How to label the paragraphs (gensim)
doc2vec model gets its algorithm from word2vec.
In word2vec there is no need to label the ...
- 1,924
11
votes
What is Hellinger Distance and when to use it?
Hellinger distance is a metric to measure the difference between two probability distributions. It is the probabilistic analog of Euclidean distance.
Given two probability distributions, $P$ and $Q$, ...
- 16.5k
11
votes
Clustering with cosine similarity
First, every clustering algorithm is using some sort of distance metric. Which is actually important, because every metric has its own properties and is suitable for different kind of problems.
You ...
- 1,641
10
votes
How to do postal addresses fuzzy matching?
As you are using R you might want to look into the stringdist package and the Jaro-Winkler distance metric that can be used in the calculations. This was developed at the U.S. Census Bureau for ...
- 720
10
votes
Accepted
What is the difference between a hashing vectorizer and a tfidf vectorizer
The main difference is that HashingVectorizer applies a hashing function to term frequency counts in each document, where ...
- 1,558
9
votes
Accepted
R error using package tm (text-mining)
You have to tell Corpus what kind of source you are using. Try:
Corpus(VectorSource(d1$Yes))
- 485
9
votes
Preference Matching Algorithm
My first suggestion would be to somehow map the non-quantifiable attributes to quantities with the help of suitable mapping functions. Otherwise, simply leave them out.
Secondly, I don't think that ...
- 1,516
9
votes
applying word2vec on small text files
Word2Vec isn't a good choice for a dataset of such size. From researches I have seen, it will unleash its power if you feed at least couple of million of words, 3k tweets wouldn't be enough for a ...
- 779
9
votes
What is the difference between NLP and text mining?
I have had this doubt since a long time. So, this post here helped me figure the differences between the two.
So, this is the difference between text mining and NLP:
Text Mining deals with the ...
- 8,016
8
votes
Accepted
Difference between tf-idf and tf with Random Forests
Decision trees (and hence Random Forests) are insensitive to monotone transformations of input features.
Since multiplying by the same factor is a monotone transformation, I'd assume that for Random ...
- 2,750
8
votes
Which classification algorithms to try for classifying text data into 300 categories
In general, a decent starting point for problems like these is Naive Bayes (NB) classification using a simple bag of words model. Here are some slides describing NB as applied to natural language ...
- 681
Only top scored, non community-wiki answers of a minimum length are eligible