1
$\begingroup$

I am a newbie in machine learning.I have 100 text documents.I need to build a model on those 100 text documents and if i give new documents it has to give whether this new document is similar to those 100 documents or not.Which is the best model for this problem?

$\endgroup$
  • $\begingroup$ If you want a binary response you need training data for a classifier, otherwise you can return the similarity according to some embedding. Even then training data helps because similarity is subjective. $\endgroup$ – Emre Jul 11 '17 at 17:44
1
$\begingroup$

Only way to check which is best model is to try different in practise and compare. Do you have any documents to train with that are not similar? Do you have labels or topics assigned to these 100 ?

Assuming you have some irrelevant documents you can use a "spam" approach to this type of problem.

I would begin with Bag of Words representing of your documents for a Naive Bayes model generation (common, good performing, baseline here) - this link http://sebastianraschka.com/Articles/2014_naive_bayes_1.html#naive-bayes-and-text-classification is a good explanation of this for documents .

I would also try using a library like Python gensim that will allow you to try its various methods, and you could try that library's Doc2Vec method. Then train a linear discriminant classifier like Log. Regression with the document vectors it gives you, and try see if its resulting accuracy is better than the Naive Bayes method.

| improve this answer | |
$\endgroup$
1
$\begingroup$

The simplest method to test if a new document is like previous documents is to hash them and look for collisions. Hashing items put them into "buckets". If two items are in the same "bucket" (aka, collide), they are similar.

Generally, people use locality-sensitive hashing (LSH) for document similarity.

The process is:

  1. Shingling - Pick a window size and slide the window over a document
  2. MinHash - Compute the hashes of the text shingles a number of times and pick the minimum to reduce data size
  3. LSH - Bucket items with the same MinHash values

Alternatively, you can compute the Jaccard Similarity between hashed documents and choose a similarity threshold.

The algorithm is covered in detail in Chapter 3/"Finding Similar Items" of Mining of Massive Datasets book.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.