Stochastic gradient descent in matrix factorization, sensitive to label's scale?

Question

I'm trying to figure out a strange phenomenon, when I use matrix factorization (the Netflix Prize solution) for a rating matrix:

$R = P^T * Q + B_u + B_i$

with ratings ranging from 1 to 10.

Then I evaluate the model by each label's absolute mean average error in test set, the first column is origin_score, the second(we don't transform the data, then train and its prediction error), the third(we transform the data all by dividing 2, train, and when I use this model to make prediction, firstly reconstruct the matrix and then just multiply 2 and make it back to the same scale)

As you see, in grade 3-4 (most samples are label from 3-4), it's more precise while in high score range(like 9 and 10, just 2% of the whole traiing set), it's worse.

+----------------------+--------------------+--------------------+
| rounded_origin_score | abs_mean_avg_error | abs_mean_avg_error | +----------------------+--------------------+---------------------+
| 1.0 | 2.185225396100167 | 2.559125413626183 | | 2.0 | 1.4072212825108161 | 1.5290497332538155 | | 3.0 | 0.7606073396581479 | 0.6285151230269825 | | 4.0 | 0.7823491986435621 | 0.6419077576969795 | | 5.0 | 1.2734369551159568 | 1.256590210555053 | | 6.0 | 1.9546560495715863 | 2.0461809588933835 | | 7.0 | 2.707229888048017 | 2.8866856489147494 | | 8.0 | 3.5084244741417137 | 3.7212155956153796 | | 9.0 | 4.357185793060213 | 4.590550124054919 | | 10.0 | 5.180752400467891 | 5.468600926567884 | +----------------------+--------------------+---------------------+

I've re-train the model several times, and got same result, so I think it's not effect by randomness.

Answer 1

The larger your target scores, the larger latent variables should be (well, it's not only a magnitude that matters, but also a variance, but it still applies to your case). There's no problem with larger coefficients of latent vectors unless you use regularization (and, likely, you do). In case of regularization your optimal solution will tend towards smaller values, and'd sometimes prefer to sacrifice some accuracy for lower regularization penalty.

Gradient Descent doesn't suffer from problem of large coefficients (unless you run into some sort of numerical issues): if the learning rate is tuned properly (there are lots of stuff on it, google), it should arrive to equivalent parameters. Otherwise nobody guarantees you convergence :-)

The common rule of thumb when doing regression (and your instance of matrix factorization is a kind of regression) is to standardize your data: make it having zero mean and unit variance.