3
$\begingroup$

The idea is to motivate the SVD for use in a recommender system.

Consider a matrix $A\in \mathbb{R}^{f\times u}$ where $A_{ij}$ caputures how user $j$ rates film $i$ (on a scale from 1-10, some entries may be missing).

Considering matrices $K=AA^T$, $L = A^TA$ what do $K_{ij}$ and $L_{ij}$ tell us?

What interpretations can you give for matrices $U$ and $V$ in the SVD $A=U\Sigma V^T$?

Riiight ... so

K = $AA^T$ = $U\Sigma^2U^T$, L = $A^TA = V\Sigma^2V^T$

With $K_{ij} = \langle A[i,:], A[j,:] \rangle$ being the dot product of all ratings for film $i$ with all ratings for film $j$.

Similarly, $L_{ij} = \langle A[:, i], A[:,j] \rangle$ being the dot product of all ratings from user $i$ with all ratings from user $j$.

That tells us ... what exactly? I expect it to amount to some kind of similarity measure, but beyond that, I have no idea.

As for $U$ and $V$ ... I know the first $r$ of them to be Eigenvectors of $K$ and $L$ for some $r$. I also know

$u_1,..,u_r$ is an orthonormal basis for the column space

$v_1,..v_r$ is an orthonormal basis for the row space

but that doesn't really tell me anything.

Yet have no idea if the notion of "row space" and "column space" even makes sense. What would the "span" of $A$'s columns / rows even mean?

I also have no idea what the nullspace of $A$ or $A^T$ would represent - if it represents anything at all - nor what

$u_{r+1},..,u_{m}$ being an orthonormal basis for $ker(A)$

or $v_{r+1},..,v_{n}$ being an orthonormal basis for $ker(A^T)$

implies - again, if anything at all.

$\endgroup$

1 Answer 1

4
$\begingroup$

First of all, note that the dot product between two movies/users is by definition the correlation between them. So your intuition for treating it as a similarity measure is not wrong.

Now, applying SVD to $A$ is simply a re-representation of it in some other basis. Let's call it feature space. If we'll take it one step further and keep only $k$ leading vectors of $U$ and $V$, we'll get two matrices: $U^{(k)} \in \mathbb{R}^{f,k}$ and $V^{(k)} \in \mathbb{R}^{k,u}$ (where $f$ and $u$ are the dimensions you supplied in your question). where $U^{(k)}\cdot V^{(k)} \approx A^{(k)}$ is a $k$-rank approximation for $A$ ** (note it has the same dimensions).

So you can of $U^{(k)}$ as a matrix which rows movies, and each movie has some $k$ latent features. Similarly, $V^{(k)}$ will be a "list" (columns) of users and each user is described by some $k$ values (latent features).

Note that you can "split" the scaling matrix $\Sigma^{(k)}$ between $U^{(k)}$ and $V^{(k)}$: $U'^{(k)} = U^{(k)}\Sigma^{(k)0.5}$ and $V'^{(k)} = V^{(k)}\Sigma^{(k)0.5}$, to make up for the lack of scale of $U$ and $V$.

This factorization can be elaborated into more interpretable methods such as LDA.

** The variational approach to singular values tell us that this approximation minimizes the difference between $A$ and any other $k$ rank matrix in terms of Frobenius-norm and operator-norm (and possibly other norms I'm not aware of).

$\endgroup$
2
  • $\begingroup$ Thank you! So if I understand you correctly that $U^{(k)}$ will infer values for some unknown characteristics all films share and that $V^{(k)}$ does - in much the same way - profile the users, do the matrices $U'^{(k)}$, $V'^{(k)}$ represent anything? Or asked differently, what is the interpretation of the singular values? $\endgroup$
    – User1291
    Mar 14, 2017 at 14:39
  • $\begingroup$ @User1291, Gladly! The singular values has no interpretation that I know of. However, they can be used to determine a good $k$. Many natural problems have few very high singular values and then they drop drastically. You can use that "drop" in order to determine the $k$ (at least that's one way to do it). Small but non-zero singular values can be thought as occurring due to noise. So that way you can use your $k$-approximation for sort of noise canceling. $\endgroup$
    – ehudk
    Mar 14, 2017 at 15:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.