A weird result from a recommender system

Question

Say there're the top 10 most popular items among 100 sales products and about 100k users regularly purchase items on daily basis.

A = has been purchased by 100k users. 
B = has been purchased by 30k users.
C = has been purchased by 20k users.
D = has been purchased by 18k users.
E = has been purchased by 10k users.
F = has been purchased by 8k users.
G = has been purchased by 7k users.
H = has been purchased by 4k users.
I = has been purchased by 3k users.
J = has been purchased by 1k users.

X = never bought by anyone.
Y = never bought by anyone.
Z = never bought by anyone.

So basing on this fact, the training data is going to have more than 50m rows of data like this.

User Id  |  User Name  |  Item Id  |  Item Name  |  label  |  Purchase Date  |
1           Thomas        1           A             true      12, Mar 2019
1           Thomas        1           A             true      13, Mar 2019
1           Thomas        1           A             true      14, Mar 2019
1           Thomas        1           A             true      15, Mar 2019
1           Thomas        2           B             true      11, Mar 2019
1           Thomas        3           C             true      09, Mar 2019
1           Thomas        4           D             true      07, Mar 2019
2           Angelica      1           E             true      12, Mar 2019
.
.
.

The preferences of users will be like this, they might be countless but let's take one example.

Thomas bought A, B, C, D
Angelica bought A, B, C, D
Gloria bought A, B, C, D
Jennifer bought A, B, C, D and I

Using the user based collaborative filtering, it is quite obvious that Thomas, Angelica, Gloria are likely to get the item I as a recommended item because Jennifer likes I item and also has the exact same purchase pattern as the others do.

With this in mind, I was starting to think that if I have another two users who bought the unpopular items X,Y,Z, the predictions on them will result in recommending the unsold items.

So I added dummy data manually before training the model like this.

User Id  |  User Name  |  Item Id  |  Item Name  |  label  |  Purchase Date  |
1           Thomas        1           A             true      12, Mar 2019
1           Thomas        1           A             true      13, Mar 2019
1           Thomas        1           A             true      14, Mar 2019
1           Thomas        1           A             true      15, Mar 2019
1           Thomas        2           B             true      11, Mar 2019
1           Thomas        3           C             true      09, Mar 2019
1           Thomas        4           D             true      07, Mar 2019
2           Angelica      1           E             true      12, Mar 2019
.
.
.
100001      Andrew        24          X             true      19, Mar 2019
100001      Andrew        25          Y             true      19, Mar 2019
100002      Andy          24          X             true      19, Mar 2019
100002      Andy          25          Y             true      19, Mar 2019
100002      Andy          26          Z             true      19, Mar 2019

As I mentioned above, I thought Andrew will get Z as a recommended item because Andrew has a common in the item preference with Andy and he bought Z as well, even if the purchase data for X,Y and Z has a extremely small portion of training data ( only 5 records exist among the 10M records of data ).

But the result was totally unexpected.

Every users have X, Y, Z in their recommended list, although the prediction score is very low compared to the others and what's more puzzling is that Andrew and Andy have no outstanding scores on the unpopular items even if they actually bought them!

I don't know why this happens, do I misunderstand the user based collaborative filtering concept?

Answer 1

Increasing the latent dimension value was the key here.

My recommendation system was implemented via ML.NET. And the framework's default setting for the latent dimension value was 20, which seems pretty small considering the volume of my training data.

Increasing the hidden feature count makes my system perform better, successfully predicting the X,Y,Z items as false candidates for other existing users except Andy and Andrew. Below is how to set the value. This is based on the example code in ML.NET

var pipeline = mlContext.Transforms.Text.FeaturizeText(outputColumnName: "userIdFeaturized", inputColumnName: nameof(MovieRating.userId))
       .Append(mlContext.Transforms.Text.FeaturizeText(outputColumnName: "movieIdFeaturized", inputColumnName: nameof(MovieRating.movieId))
       .Append(mlContext.Transforms.Concatenate(DefaultColumnNames.Features, "userIdFeaturized", "movieIdFeaturized"))
       .Append(mlContext.BinaryClassification.Trainers.FieldAwareFactorizationMachine(
           new string[] {DefaultColumnNames.Features}
           , (e) => { e.latentDim = 200; }) // set custom value here.
           )
       );

To my best knowledge

When decomposing matrices, SVD extracts hidden features from the matrix of the training data and the hidden layers will be directly related to each users and a set of items in the middle of each entities, which is referred as dimension reduction here. I guess too small latent dimension generalizes the variety of recommendation items. That's why, I reckon, increasing the value solves my problem.

Any correction or comment would be appreciated. I definitely don't want to deliver false belief.