How to encode an array of categories to feed into sklearn

Question

I'm working on a recommendation problem, broadly following the Youtube paper on theirs. Their surrogate problem is to recommend the next video a user will watch. One feature they include in their model is a representation of watch history (I.E. which videos a user has watched), which they achieve by embedding the Video ID's and then averaging that embedding.

So for data, they might have...

[
    [1234, 5678, 9012],
    [1234],
    [9012, 1234, 8245],
]

and each of those IDs is embedded into say 8 features, those 8 features are averaged for every observation in the training data and those 8 average features are input to the deep layers of the network.

Now I can replicate that in keras using a combination of tf.keras.preprocessing.sequence.pad_sequences plus Embedding() and GlobalAveragePooling1D(), and that works reasonably well. My problem is that I'd like to evaluate other algorithms on this same problem (after all, I have WAY less data than Youtube does so I'm not sure I need an NN at all). I can't really conceptualise how to deal with this encoding of historical "watches" though in a way that would work in a more traditional algorithm like XGBoost or RandomForests in sklearn. Ordinal Encoding and taking the average seems plain silly, I obviously can't OneHot encode the 1st to nth video watch and so on. Any ideas of how to tackle this would be greatly appreciated.

EDIT:

I guess possible FeatureHasher fits the bill? Rather than an average, it produces a sum of each of the hashed input categories. That's still (be default) gonna lead to 10s of thousands of features though so I'm not sure that it's going to be a significant improvement over OneHotEncoding!

EDIT 2:

Alternatively, perhaps I could treat this as a text embedding problem by concatenating the array of categories into a single string and use CountVectorizer to create an array of (n_samples, n_categories), then use TruncatedSVD to reduce that to a more manageable number of components.

I guess I've a couple of ideas there, anyone know if they're worth trying?

Answer 1

If the sequence of watches is meaningful, then you do need some kind of classifier that creates user/item embeddings from the sequence of watches, so you're probably looking at GRUs and LSTMs. Neural nets aren't overkill here per se. This is a pretty good treatment of the topic: https://towardsdatascience.com/introduction-to-recommender-system-part-2-adoption-of-neural-network-831972c4cbf7

However this simple approach above does mean you have a multi-label class with a lot of dimensions. This can be hard to scale and may not train well without much data. Feature hashing doesn't help so much in the output because you need to map each back to the right video

If you're willing to forego the sequence, which often doesn't carry that much information beyond simple recency (recent watches are more important). Yes the usual way of dealing with this is as a huge matrix factorization problem. Sparse SVD is possible but maybe overkill; ALS is simpler and about as effective, and works well for implicit input like you have here. You can readily add weights to watches with ALS, which can be used to deemphasize older watches.

Answer 2

So in the end I dealt with this as a latent semantic analysis problem by concatenating the array of category descriptions into a single long string, then passing that through sklearn's TfIdfTransformer and TruncatedSVD. This worked fine, although the neural network I had already built outperformed any sklearn algorithm I tried.