51
There's a number of different ways of going about this depending on exactly how much semantic information you want to retain and how easy your documents are to tokenize (html documents would probably be pretty difficult to tokenize, but you could conceivably do something with tags and context.)
Some of them have been mentioned by ffriend, and the paragraph ...
34
Text Classification
I give you a bunch of documents, each of which has a label attached. I ask you to learn why you think the contents of the documents have been given these labels based on their words. Then I give you new documents and ask what you think the label for each one should be. The labels have meaning to me, not to you necessarily.
Topic ...
31
Shallow Natural Language Processing technique can be used to extract concepts from sentence.
-------------------------------------------
Shallow NLP technique steps:
Convert the sentence to lowercase
Remove stopwords (these are common words found in a language. Words like for, very, and, of, are, etc, are common stop words)
Extract n-gram i.e., a ...
25
What you're describing is often achieved using a simple combination of TF-IDF and extractive summarization.
In a nutshell, TF-IDF tells you the relative importance of each word in each document, in comparison to the rest of your corpus. At this point, you have a score for each word in each document approximating its "importance." Then you can use these ...
16
Personally I would advocate using something that is both not-specific to the NLP field, and something that is sufficiently general that it can still be used as a tool even when you've started moving beyond this level of metadata. I would especially pick a format that can be used regardless of development environment and one that can keep some basic structure ...
16
The Google Research Blog should be helpful in the context of TensorFlow.
In the above article, there is a reference to the Annotated English Gigaword dataset which is routinely used for text summarization.
The 2014 paper by Sutskever et al titled Sequence to Sequence Learning with Neural Networks could be a meaningful start on your journey as it turns out ...
16
This is an open area of research and it certainly depends on the way you frame the problem. If you're talking about multi-document summarization then the problem is slightly different than if you were talking about single-document summarization.
It's worth briefly reviewing the literature.
The link provided by u/Society Of Data Scientists is great and it'...
15
The word2vec algorithm may be a good way to retrieve more elements for a list of similar words. It is an unsupervised "deep learning" algorithm that has previously been demonstrated with Wikipedia-based training data (helper scripts are provided on the Google code page).
There are currently C and Python implementations. This tutorial by Radim Řehůřek, the ...
14
Some standard datasets for text classification are the 20-News group, Reuters (with 8 and 52 classes) and WebKb. You can find all of them here.
14
Check out this link.
Here, they will take you through loading unstructured text to creating a wordcloud. You can adapt this strategy and instead of creating a wordcloud, you can create a frequency matrix of terms used. The idea is to take the unstructured text and structure it somehow. You change everything to lowercase (or uppercase), remove stop words, ...
14
Let's work it out from the ground up. Classification (also known as categorization) is an example of supervised learning. In supervised learning you have:
model - something that approximates internal structure in your data, enabling you to reason about it and make useful predictions (e.g. predict class of an object); normally model has parameters that you ...
13
Here are the basic Natural Language Processing capabilities (or annotators) that are usually necessary to extract language units from textual data for sake of search and other applications:
Sentence breaker - to split text (usually, text paragraphs) to sentences. Even in English it can be hard for some cases like "Mr. and Mrs. Brown stay in room no. 20."
...
13
But I don't know what is difference between text classification and topic models in documents
Text Classification is a form of supervised learning, hence the set of possible classes are known/defined in advance, and won't change.
Topic Modeling is a form of unsupervised learning (akin to clustering), so the set of possible topics are unknown apriori. They'...
13
Check the Stanford NLP Group's open source software, in particular, Stanford Classifier. The software is written in Java, which will likely delight you, but also has bindings for some other languages. Note, the licensing - if you plan to use their code in commercial products, you have to acquire commercial license.
Another interesting set of open source ...
13
During NLP and text analytics, several varieties of features can be extracted from a document of words to use for predictive modeling. These include the following.
ngrams
Take a random sample of words from words.txt. For each word in sample, extract every possible bi-gram of letters. For example, the word strength consists of these bi-grams: {st, tr, re, ...
12
I agree with Sean's answer.
NLP and text mining are usually used for different goals.
Also, there is indeed an overlap and both definitions are vogue.
Other than the difference in goal, there is a difference in methods.
Text mining techniques are usually shallow and do not consider the text structure. Usually, text mining will use bag-of-words, n-grams and ...
12
Both are possible. You can give every document a unique ID (such as a sequential serial number) as a doctag, or a shared string doctag representing something else about it, or both at the same time.
The TaggedDocument constructor takes a list of tags. (If you happen to limit yourself to to plain ints ascending from 0, the Doc2Vec model will use those as ...
12
I'm not sure Topic Modelling will help you here, as it tries to extract abstract topics from text. I'm afraid resumes might be too 'dry' for it to work nicely.
Here are a few sources I found that might be helpful:
A resume parser
The reply to this post, that gives you some text mining basics (how to deal with text data, what operations to perform on it, ...
11
Just extract keywords and train a classifier on them. That's all, really.
Most of the text in CVs is not actually related to skills. E.g. consider sentence "I'm experienced and highly efficient in Java". Here only 1 out of 7 words is a skill name, the rest is just a noise that's going to put your classification accuracy down.
Most of CVs are not really ...
11
doc2vec model gets its algorithm from word2vec.
In word2vec there is no need to label the words, because every word has their own semantic meaning in the vocabulary.
But in case of doc2vec, there is a need to specify that how many number of words or sentences convey a semantic meaning, so that the algorithm could identify it as a single entity. For this ...
11
Hellinger distance is a metric to measure the difference between two probability distributions. It is the probabilistic analog of Euclidean distance.
Given two probability distributions, $P$ and $Q$, Hellinger distance is defined as:
$$h(P,Q) = \frac1{\sqrt2}\cdot \|\sqrt{P}-\sqrt{Q}\|_2$$
It is useful when quantifying the difference between two ...
11
First, every clustering algorithm is using some sort of distance metric. Which is actually important, because every metric has its own properties and is suitable for different kind of problems.
You said you have cosine similarity between your records, so this is actually a distance matrix. You can use this matrix as an input into some clustering algorithm.
...
10
As you are using R you might want to look into the stringdist package and the Jaro-Winkler distance metric that can be used in the calculations. This was developed at the U.S. Census Bureau for linking .
See for more information on the Jaro and Jaro-Winkler distance in this journal.
For a comparison of different matching techniques, read this paper
10
The main difference is that HashingVectorizer applies a hashing function to term frequency counts in each document, where TfidfVectorizer scales those term frequency counts in each document by penalising terms that appear more widely across the corpus. There’s a great summary here.
Hash functions are an efficient way of mapping terms to features; it doesn’t ...
9
You have to tell Corpus what kind of source you are using. Try:
Corpus(VectorSource(d1$Yes))
9
My first suggestion would be to somehow map the non-quantifiable attributes to quantities with the help of suitable mapping functions. Otherwise, simply leave them out.
Secondly, I don't think that you need to assume that the list of attributes is not finite. A standard and intuitive approach is to represent each attribute as an individual dimension in a ...
9
Word2Vec isn't a good choice for a dataset of such size. From researches I have seen, it will unleash its power if you feed at least couple of million of words, 3k tweets wouldn't be enough for a concise word similarity.
9
I have had this doubt since a long time. So, this post here helped me figure the differences between the two.
So, this is the difference between text mining and NLP:
Text Mining deals with the text itself, while NLP deals with the
underlying/latent metadata.
Answering questions like - frequency counts of words, length of the
sentence, presence/...
8
In general, a decent starting point for problems like these is Naive Bayes (NB) classification using a simple bag of words model. Here are some slides describing NB as applied to natural language processing. There's nothing especially fancy about this approach, but it's pretty easy to implement and will give you a starting point to expand from.
Once you've ...
8
There is a nice implementation of this in gensim: http://radimrehurek.com/gensim/models/phrases.html
Basically, it uses a data-driven approach to detect phrases, ie. common collocations. So if you feed the Phrase class a bunch of sentences, and the phrase "big data" comes up a lot, then the class will learn to combine "big data" into a single token "...
Only top voted, non community-wiki answers of a minimum length are eligible