SVD for recommendation engine

Question

I'm trying to build a toy recommendation engine to wrap my mind around Singular Value Decomposition (SVD). I've read enough content to understand the motivations and intuition behind the actual decomposition of the matrix A (a user x movie matrix).

I need to know more about what goes on after that.

from numpy.linalg import svd
import numpy as np

A = np.matrix([ 
  [0, 0, 0, 4, 5],
  [0, 4, 3, 0, 0],
  ...
])
U, S, V = svd(A)

k = 5 #dimension reduction
A_k = U[:, :k] * np.diag(S[:k]) * V[:k, :]

Three Questions:

1. Do the values of matrix A_k represent the the predicted/approximate ratings?

2. What role/ what steps does cosine similarity play in the recommendation?

3. And finally I'm using Mean Absolute Error (MAE) to calculate my error. But what I'm values am I comparing? Something like MAE(A, A_k) or something else?

Answer 1

You can use SVD to build a recommendation engine, but I don't think it's the best way to get intuition around what's going on under the hood. Regardless, here's a presentation with more details, I'd recommend reviewing slide 9.

And to answer your questions:

1. A_k represents an embedding dimension (i.e. the low-rank approximation) that is used to predict the user-rating matrix.

2. The cosine similarity is just the dot product for user $i$ and item $j$, which maps to the predicted rating for user $i$ and item $j$. The dot product is what defines the users and items as being similar.

3. Yes, you should use the MAE on A and A_k. You may prefer to use MSE instead. This measures the quality of your predictions for user $i$ and item $j$. Note, this is obviously the MSE of a matrix, which is the Frobenius Norm.

I think an easier way to understand SVD is to see it applied to image compression for different components. See this presentation here.

Answer 2

Let me try to answer your questions.

1. The $a^{(k)}_{ij}$ element of the matrix $A_k$ is an approximation of the $a_{ij}$ element of the original matrix $A$. In particular, if you were to use the $U$, $\Sigma$, and $V$ matrices of expansion as they are (without the dimension reduction) you would have $a^{(k)}_{ij} = a^{ij}$. As a corollary, it's impossible to use the $A_k$ as a prediction matrix, because you already have this information in matrix $A$. For example, if the $p^{th}$ user did not rate the $q^{th}$ product, $a_{pq}$ will be $0$. At the same time $a^{(k)}_{pq}$ will be $0$ too. Not a very insightful prediction, is it?

2. Cosine similarity allows you to judge if two vectors have the same orientation, in other words to what extent are they similar. Now let's look at the components of the SVD decomposition once again. $V$ defines the hidden product's features and the $i^{th}$ column of the matrix $V$ addresses the $i^{th}$ product. $U$ defines how a user reacts to the presence or absence of a product's particular feature. Knowing the features of the products you can find similar products using the cosine similarity. For example, the similarity of the 1st and the 2nd products is simply the cosine between the $V$'s first and second columns.

3. You should use the mean absolute error or the mean squared error multiple times while implementing the recommendation engine. First, if you want to perform dimension reduction you will need to find $k$ for which the value of $\mbox{MEAN}(A, A_k)$ is reasonable. The second case for MEAN will be described below.

Interim summary: you shouldn't use $A_k$ for predictions, instead you should use the components of expansion themselves to find similar products. Now let's formulate said above as an algorithm that implements a naive recommendation engine using SVD.

1. Decompose the original matrix $A$ into three matrices $U$, $\Sigma$, and $V$.
2. Knowing the hidden features $V$, for each product find its similarity with all other products.
3. For each user find products he or she has already rated high. Pick the necessary quantity of products similar to those the user likes and return them as the recommendations.

If you split your data into train and test sets, you can examine your predictions at the moment by calculating the MEAN between what your engine predictions are for the test set and how users actually rated products from it.

This schema can be inverted in the sense of you can find similar users instead of similar products and recommend products similar users like.

Hope this will help.