Interpretation of an SVD for recommender systems

Question

The idea is to motivate the SVD for use in a recommender system.

Consider a matrix $A\in \mathbb{R}^{f\times u}$ where $A_{ij}$ caputures how user $j$ rates film $i$ (on a scale from 1-10, some entries may be missing).

Considering matrices $K=AA^T$, $L = A^TA$ what do $K_{ij}$ and $L_{ij}$ tell us?

What interpretations can you give for matrices $U$ and $V$ in the SVD $A=U\Sigma V^T$?

Riiight ... so

K = $AA^T$ = $U\Sigma^2U^T$, L = $A^TA = V\Sigma^2V^T$

With $K_{ij} = \langle A[i,:], A[j,:] \rangle$ being the dot product of all ratings for film $i$ with all ratings for film $j$.

Similarly, $L_{ij} = \langle A[:, i], A[:,j] \rangle$ being the dot product of all ratings from user $i$ with all ratings from user $j$.

That tells us ... what exactly? I expect it to amount to some kind of similarity measure, but beyond that, I have no idea.

As for $U$ and $V$ ... I know the first $r$ of them to be Eigenvectors of $K$ and $L$ for some $r$. I also know

$u_1,..,u_r$ is an orthonormal basis for the column space

$v_1,..v_r$ is an orthonormal basis for the row space

but that doesn't really tell me anything.

Yet have no idea if the notion of "row space" and "column space" even makes sense. What would the "span" of $A$'s columns / rows even mean?

I also have no idea what the nullspace of $A$ or $A^T$ would represent - if it represents anything at all - nor what

$u_{r+1},..,u_{m}$ being an orthonormal basis for $ker(A)$

or $v_{r+1},..,v_{n}$ being an orthonormal basis for $ker(A^T)$

implies - again, if anything at all.

Answer 1

First of all, note that the dot product between two movies/users is by definition the correlation between them. So your intuition for treating it as a similarity measure is not wrong.

Now, applying SVD to $A$ is simply a re-representation of it in some other basis. Let's call it feature space. If we'll take it one step further and keep only $k$ leading vectors of $U$ and $V$, we'll get two matrices: $U^{(k)} \in \mathbb{R}^{f,k}$ and $V^{(k)} \in \mathbb{R}^{k,u}$ (where $f$ and $u$ are the dimensions you supplied in your question). where $U^{(k)}\cdot V^{(k)} \approx A^{(k)}$ is a $k$-rank approximation for $A$ ** (note it has the same dimensions).

So you can of $U^{(k)}$ as a matrix which rows movies, and each movie has some $k$ latent features. Similarly, $V^{(k)}$ will be a "list" (columns) of users and each user is described by some $k$ values (latent features).

Note that you can "split" the scaling matrix $\Sigma^{(k)}$ between $U^{(k)}$ and $V^{(k)}$: $U'^{(k)} = U^{(k)}\Sigma^{(k)0.5}$ and $V'^{(k)} = V^{(k)}\Sigma^{(k)0.5}$, to make up for the lack of scale of $U$ and $V$.

This factorization can be elaborated into more interpretable methods such as LDA.

** The variational approach to singular values tell us that this approximation minimizes the difference between $A$ and any other $k$ rank matrix in terms of Frobenius-norm and operator-norm (and possibly other norms I'm not aware of).