For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task.
Item similarity approach is just about creating a square matrix of items in your catalog (i.e. itemID X itemID), where each element of the matrix is the magnitude of similarity between and . This magnitude of similarity can be calculated by using any similarity function, most popular being the Cosine Similarity.
In spark this can be done by:
- Create a
k X n matrix, where n items are described as k-dimensioned vectors. For representing each item as a
k dimension vector, you can use
ALS which represents each entity in a latent factor space. The dimension of this space (
k) can be chosen by you.
This k X n matrix can be represented as
- Convert this k X n matrix to RowMatrix.
columnSimilarities() function to get a
n X n matrix of similarities between n items.
Finally , whenever an itemi is being viewed, you can recommend other
m items obtained by sorting the items in row i by the decreasing value of similarities and picking top
More details can be found here.