25

At the expense of over-simplication, latent features are 'hidden' features to distinguish them from observed features. Latent features are computed from observed features using matrix factorization. An example would be text document analysis. 'words' extracted from the documents are features. If you factorize the data of words you can find 'topics', where '...


21

Item Based Algorithm for every item i that u has no preference for yet for every item j that u has a preference for compute a similarity s between i and j add u's preference for j, weighted by s, to a running average return the top items, ranked by weighted average User Based Algorithm for every item i that u has no preference for yet for ...


17

Here some resources that might be helpful: Recommenderlab - a framework and open source software for developing and testing recommendation algorithms: http://lyle.smu.edu/IDA/recommenderlab. Corresponding R package recommenderlab: http://cran.r-project.org/package=recommenderlab. The following blog post illustrates the use of recommenderlab package (which ...


14

Depends on the user's intent, for starters. Users normally only view the first set of links, which means that unless the link is viewable, it's not getting clicks; meaning you'd have to be positive those are the best links, otherwise the clicks are most likely going to reflect placement, not relevance. For example, here's a click and attention distribution ...


11

You are correct that both models work on the same data without any problem. Both items operate on a matrix of user-item ratings. In the user-based approach the algorithm produces a rating for an item i by a user u by combining the ratings of other users u' that are similar to u. Similar here means that the two user's ratings have a high Pearson correlation ...


10

You are doing the correct thing. Technically, this averaging leads to computing the centroid in the Euclidean space of a set of N points. The centroid works pretty well with cosine similarities (cosine of the angles between normalized vectors), e.g. the Rocchio algorithm.


10

Short answer: Cosine distance is not the overall best performing distance metric out there Although similarity measures are often expressed using a distance metric, it is in fact a more flexible measure as it is not required to be symmetric or fulfill the triangle inequality. Nevertheless, it is very common to use a proper distance metric like the Euclidian ...


9

My first suggestion would be to somehow map the non-quantifiable attributes to quantities with the help of suitable mapping functions. Otherwise, simply leave them out. Secondly, I don't think that you need to assume that the list of attributes is not finite. A standard and intuitive approach is to represent each attribute as an individual dimension in a ...


9

Lots of questions here. First, for a truly new user with no data, there is no way to use a recommender model. If you have literally no information on the user, the only thing you can do is provide some default recommendations. Of course, once you have any data, and you can rebuild the model to incorporate the user, you can make recommendations. You can do ...


8

nDCG is used to evaluate a golden ranked list (typically human judged) against your output ranked list. The more is the correlation between the two ranked lists, i.e. the more similar are the ranks of the relevant items in the two lists, the closer is the value of nDCG to 1. RMSE (Root Mean Squared Error) is typically used to evaluate regression problems ...


8

Suppose you have (MxN) sparse matrix, where M -- stands for number of users who gave recommendations, and N is the number of items recommended. The $x_{ij}$ element of the matrix is the recommendation given, with some elements missing, i.e. to be predicted. Then your matrix can be "factorized", via introducing K "latent factors", so that instead of one ...


8

The obvious answer would be the Netflix prize dataset, there is a lot of research into it and most CF algorithms have known scores in it. There are other available datasets that are usually used as benchmarks: Movie lens Dataset: a 20 million ratings dataset used for benchmarking CF algorithms; Jester Dataset: a joke recommendation dataset with more than ...


8

There's nothing about a recommendation system that absolutely necessitates some kind of machine learning. Indeed, I've seen decision systems in use that were essentially just someone's idea about what the customer's preferences ought to be. A recommender can be based on anything from a few ad-hoc 'common sense' rules, to a logistic regression someone did on ...


7

As suggested, going wild. First of all, correct me if I’m wrong: Just a few features exist for each unique product; There is no ultimate features list, and clients are able to add new features to their products. If so, constructing full product-feature table could be computational expensive. And final data table would be extremely sparse. The first step ...


7

For my part I can say that I use click frequency on i.e. eCommerce products. When you combine it with the days of the year it can even bring you great suggestions. i.e.: We have historical data from 1 year over 2 products (Snowboots[], Sandalettes[]) Snowboots[1024,1253,652,123,50,12,8,4,50,148,345,896] Sandalettes[23,50,73,100,534,701,1053,1503,1125,453,...


7

It's actually defined on the first page: ... sparsity level (ratio of observed to total ratings) ... In other words, the fraction of the user/item rating matrix that is not empty. Remember that the problem is that most user-item pairs have no rating, and we wish to estimate them. Example: Let there be three users and four products. The number of ...


7

For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task. Item similarity approach is just about creating a square matrix of items in your catalog (i.e. itemID X itemID), where each element of the matrix is the magnitude of similarity between and . This ...


6

Your system isn't just trained on items that are recommended right? if so you have a big feedback loop here. You want to learn from all clicks/views, I hope. You suggest that not-looking at an item is a negative signal. I strongly suggest you do not treat it that way. Not interacting with something is almost always best treated as no information. If you ...


6

The key is establishing a proper validation metric. I notice you talk about how you tried different recommendation algorithms, but at the end of the day you evaluated them all with RMSE. But there's no particular reason to believe that minimizing RMSE generates a "subjectively better" recommendation experience for the user - it just happens to be convenient,...


6

You could use Content based filtering but then you have to intelligently pre process the data to extract all the contents of the products. Also, that might lead to leaving a some features, This article is a great head start after you preprocess all the data. Also, you could make pseudo ratings for product vs a customer. That would depend on your problem ...


6

Here are some excerpts from the original paper that I think are key to understanding the question: Instead of using an own model parameter for each interaction, the FM models the interaction by factorizing it. We will see later on, that this is the key point which allows high quality parameter estimates of higher-order interactions under sparsity. ...


5

Is it valid to use click frequency, then yes. Is it valid to use only the click frequency, then probably no. Search relevance is much more complicated than just one metric. There are entire books on the subject. Extending this answer beyond a simple yes/no would likely make the answer far too broad (and opinionated)


5

Try exploring the rich field of "Anomaly Detection in Time Series". Control charts and CUSUMs (or cumulative sum control charts) might help you. Simple Bullet Graphs might be all you need. Based on historical data and domain knowledge, define normal variance. Then make it clear to stakeholders when the current value is outside of predefined ranges. Stephen ...


5

Instead of collaborative filtering I would use the matrix factorization approach, wherein users and movies alike a represented by vectors of latent features whose dot products yield the ratings. Normally one merely selects the rank (number of features) without regard to what the features represent, and the algorithm does the rest. Like PCA, the result is not ...


5

Rating bias and scale can easily be accounted for by standardization. The point of using Euclidean similarity metrics in vector space co-embeddings is that it reduces the recommendation problem to one of finding the nearest neighbors, which can be done efficiently both exactly and approximately. What you don't want to do in real-life settings is to have to ...


5

As You said, the most common situation for recommender system is to predict rating. Then RMSE/MAE is used. For results of a ranked item list different measures are used, e.g. Prec@K, Rec@K, AUC, NDCG, MRR, ERR. However, the are many algorithms for recommendation with its own hyper-parameters and specific use cases. Depending what You are trying to achieve ...


5

Big Picture: First of all, the feature set in your data is pretty sparse and uninteresting, so you should not expect to gain much traction from this problem. Use your human mind to think about the types of clusters that could possibly exist and this may help inform the feature engineering that you should perform. Some people may tend to buy things on ...


5

there are no product ratings available, thus collaborative filtering is not an option Wrong. You can do collaborative filtering with holdings. Just use the numbers/duration of holdings instead of ratings. That's why I want to develop recommendations on similarities between the users based on their purchase history and/or demographics Then any ...


5

You should look at the Jaccard Index, is the de facto similarity between set of items, where the sets are represented using a boolean vector. In this boolean vector each coordinate represents an item, 1 means the item is present, 0 otherwise. For example: for an universe of items banana, orange and apple. the set banana, orange will be represented by (1, 1, ...


5

Let's do this in the opposite order of how you asked. i.e. first: How can I think of the cross-product transformation in general? For me a cross-product comes from linear algebra, and it is a different transformation. Therefore let's start by not confusing both. The cross-product in the equation above is a transformation alright but is not related to ...


Only top voted, non community-wiki answers of a minimum length are eligible