I would like to build an recommendation engine using spark's Mlib itemsimilarity as mentioned here LINK
But it seems spark do not have this algorithm any more and some forums suggesting me to use ALS, but please clarify is item similarity removed from spark or any other alter natives?
I want to index my indicators into search engine and I find Sparks Item similarity best suits for me. Please advice.
For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task.
Item similarity approach is just about creating a square matrix of items in your catalog (i.e. itemID X itemID), where each element of the matrix is the magnitude of similarity between
and
. This magnitude of similarity can be calculated by using any similarity function, most popular being the Cosine Similarity.
In spark this can be done by:
k X n matrix, where n items are described as k-dimensioned vectors. For representing each item as a k dimension vector, you can use ALS which represents each entity in a latent factor space. The dimension of this space (k) can be chosen by you. This k X n matrix can be represented as RDD[Vector].
columnSimilarities() function to get a n X n matrix of similarities between n items.Finally , whenever an itemi is being viewed, you can recommend other m items obtained by sorting the items in row i by the decreasing value of similarities and picking top m.
More details can be found here.
itemFactors(model.userFactors) and userFactors(model.itemFactors), which are latent representations of each respective entity.
$\endgroup$
There is spark-itemsimilarity command line tool that is based on Spark and Mahout. (It is not a library you import inside a Spark application.) Here is an explanation of how it works.
