I would like to build an recommendation engine using spark's Mlib itemsimilarity as mentioned here LINK

But it seems spark do not have this algorithm any more and some forums suggesting me to use ALS, but please clarify is item similarity removed from spark or any other alter natives? I want to index my indicators into search engine and I find Sparks Item similarity best suits for me. Please advice.


2 Answers 2


For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task.

Item similarity approach is just about creating a square matrix of items in your catalog (i.e. itemID X itemID), where each element r_{ij} of the matrix is the magnitude of similarity between itemi and itemj. This magnitude of similarity can be calculated by using any similarity function, most popular being the Cosine Similarity.

In spark this can be done by:

  1. Create a k X n matrix, where n items are described as k-dimensioned vectors. For representing each item as a k dimension vector, you can use ALS which represents each entity in a latent factor space. The dimension of this space (k) can be chosen by you.

This k X n matrix can be represented as RDD[Vector].

  1. Convert this k X n matrix to RowMatrix.
  2. Use columnSimilarities() function to get a n X n matrix of similarities between n items.

Finally , whenever an itemi is being viewed, you can recommend other m items obtained by sorting the items in row i by the decreasing value of similarities and picking top m.

More details can be found here.

  • $\begingroup$ @Santoshi M Can you please explain more on how ALS can be used to represent entity in latent factor space? $\endgroup$
    – saurzcode
    Commented Apr 14, 2017 at 21:07
  • 1
    $\begingroup$ @saurzcode using ALS model you can get itemFactors(model.userFactors) and userFactors(model.itemFactors), which are latent representations of each respective entity. $\endgroup$
    – prafi
    Commented Jul 20, 2020 at 16:38

There is spark-itemsimilarity command line tool that is based on Spark and Mahout. (It is not a library you import inside a Spark application.) Here is an explanation of how it works.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.