Regularization term in Matrix Factorization

I'm trying to build a naive recommender system using latent factor model for MovieLens dataset. From the observed set of ratings I'm trying to build a model which will decompose the sparse matrix to N * K and K * M, where N is the number of users, M is the number of Movies and K is the number of dimensions in the latent space that I'm trying to learn. The loss function that I'm using:

$$L = \sum_{u, i}(\mathbf{x}_u^T \cdot \mathbf{y}_i + \mu + avg_u + avg_i - r_{ui})^2$$

$$r_{ui}$$ is the rating given by user $$u$$ to movie $$i$$, $$\mathbf{x}_u$$ is the latent representation of user $$u$$, $$\mathbf{y}_i$$ is the latent representation of movie $$i$$, $$\mu$$ is the average global rating, $$avg_u$$ is the average rating given by user $$u$$ and $$avg_i$$ is the average rating given for movie $$i$$. Note: $$avg$$ rating here is adjusted as per the average deviation from the global mean rating.

If I consider regularization term as

$$\lambda\sum_{u}||\mathbf{x_u}||^2 + \lambda\sum_{i}||\mathbf{y_i}||^2$$

Model will tend to penalize more commonly occurring training users/movies, therefore popular movies tend to get penalized more with this regularization term compared to the movies that are there in the long tail.

Is it fine to consider the above regularization term or is there a better way to uniformly penalize items irrespective of their occurrence in training data?

• From my understanding since you are summing up over all users $u$ and all movies $i$ in the regularization term, you are not penalizing common users/movies more than rare ones. If you sum over all user-movie rating record, you might penalize common users/movies too much, but you are not doing that. – user12075 Jan 9 at 22:02
• What if I regularize the current pair of user and item pair $(u, i)$, is it a good way to regularize embeddings? – lone_rider Jan 10 at 18:40
• That's what I thought about summing over all user-movie pairs. In that way I would worry about regularizing frequent users/movies too much. – user12075 Jan 10 at 19:43
• But on every iteration if I do regularization over all users and movies, wouldn’t that be too expensive? – lone_rider Jan 11 at 5:50