I am a newbie in machine learning.I have 100 text documents.I need to build a model on those 100 text documents and if i give new documents it has to give whether this new document is similar to those 100 documents or not.Which is the best model for this problem?
Only way to check which is best model is to try different in practise and compare. Do you have any documents to train with that are not similar? Do you have labels or topics assigned to these 100 ?
Assuming you have some irrelevant documents you can use a "spam" approach to this type of problem.
I would begin with Bag of Words representing of your documents for a Naive Bayes model generation (common, good performing, baseline here) - this link http://sebastianraschka.com/Articles/2014_naive_bayes_1.html#naive-bayes-and-text-classification is a good explanation of this for documents .
I would also try using a library like Python gensim that will allow you to try its various methods, and you could try that library's Doc2Vec method. Then train a linear discriminant classifier like Log. Regression with the document vectors it gives you, and try see if its resulting accuracy is better than the Naive Bayes method.
The simplest method to test if a new document is like previous documents is to hash them and look for collisions. Hashing items put them into "buckets". If two items are in the same "bucket" (aka, collide), they are similar.
Generally, people use locality-sensitive hashing (LSH) for document similarity.
The process is:
- Shingling - Pick a window size and slide the window over a document
- MinHash - Compute the hashes of the text shingles a number of times and pick the minimum to reduce data size
- LSH - Bucket items with the same MinHash values
Alternatively, you can compute the Jaccard Similarity between hashed documents and choose a similarity threshold.