Skip to main content
OverflowAI is here! AI power for your Stack Overflow for Teams knowledge community. Learn more
16 votes
Accepted

Can I use cosine similarity as a distance metric in a KNN algorithm

Short answer: Cosine distance is not the overall best performing distance metric out there Although similarity measures are often expressed using a distance metric, it is in fact a more flexible ...
Lejafar's user avatar
  • 296
10 votes

Spark ALS: recommending for new users

Lots of questions here. First, for a truly new user with no data, there is no way to use a recommender model. If you have literally no information on the user, the only thing you can do is provide ...
Sean Owen's user avatar
  • 6,595
8 votes
Accepted

Benchmark datasets for collaborative filtering

The obvious answer would be the Netflix prize dataset, there is a lot of research into it and most CF algorithms have known scores in it. There are other available datasets that are usually used as ...
João Almeida's user avatar
8 votes
Accepted

Do recommendation systems necessarily use machine learning algorithms?

There's nothing about a recommendation system that absolutely necessitates some kind of machine learning. Indeed, I've seen decision systems in use that were essentially just someone's idea about what ...
Ingolifs's user avatar
  • 278
8 votes
Accepted

How is the cross-product transformation defined for binary features?

Let's do this in the opposite order of how you asked. i.e. first: How can I think of the cross-product transformation in general? For me a cross-product comes from linear algebra, and it is a ...
grochmal's user avatar
  • 482
7 votes
Accepted

How do you calculate how dense or sparse a dataset is?

It's actually defined on the first page: ... sparsity level (ratio of observed to total ratings) ... In other words, the fraction of the user/item rating matrix that is not empty. Remember that ...
Emre's user avatar
  • 10.5k
7 votes

spark item similarity recommendation

For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task. Item similarity approach is just about creating ...
Santoshi M's user avatar
7 votes

precision@k and recall@k

A quick answer. Note: Checking the references I could access fully, there are no discrepancies between the definitions as long as the terms are translated correctly. Some definitions: Relevant items: ...
Nikos M.'s user avatar
  • 2,353
7 votes

What is difference between Nearest Neighbor and KNN?

Not really sure about it, but KNN means K-Nearest Neighbors to me, so both are the same. The K just corresponds to the number of nearest neighbours you take into account when classifying. Maybe what ...
Ubikuity's user avatar
  • 626
6 votes

Clustering users based on buying behaviour

Big Picture: First of all, the feature set in your data is pretty sparse and uninteresting, so you should not expect to gain much traction from this problem. Use your human mind to think about the ...
AN6U5's user avatar
  • 6,818
6 votes
Accepted

Recommender system based on purchase history, not ratings

You could use Content based filtering but then you have to intelligently pre process the data to extract all the contents of the products. Also, that might lead to leaving a some features, This ...
janpreet singh's user avatar
6 votes

Factorization Machine - prevent over fitting

Here are some excerpts from the original paper that I think are key to understanding the question: Instead of using an own model parameter for each interaction, the FM models the interaction by ...
Psidom's user avatar
  • 313
6 votes

Why is deep learning used in recommender systems?

"Recommender Systems" is a very broad area and can be approached from different optics: latent variable models, graph models, etc. "Deep learning" is an umbrella term for gradient-...
noe's user avatar
  • 26.9k
6 votes

How to evaluate when recommender systems are influencing behavior?

This is a case for A/B testing: Two versions of the website are prepared, one offering the coupon and another not offering it. Users are randomly split into two groups, some of them getting the coupon ...
Erwan's user avatar
  • 25.5k
6 votes

How to evaluate when recommender systems are influencing behavior?

As Erwan has pointed out, the straightforward way to tackle this problem is using A/B testing, so I'll point out another technique to solve the issue, which is counterfactual evaluation. In ...
alexmolas's user avatar
  • 556
6 votes

How to build a personalized recommendation system using real life scenario data?

100k apps, 300k users - quite the task. Modern ranking systems typically consist of 3 phases. Recall (Generate candidates) For each user, reduce the number of applications you could recommend them ...
GooJ's user avatar
  • 435
5 votes
Accepted

Which supervised learning algorithms are available for matching?

You can try to frame this problem as a recommender systems situation. Where you have your users (prospective students) and items (alumni) and want to recommend to the users one item. It's not a ...
João Almeida's user avatar
5 votes

Meaning of latent features?

Another example, consider the case of users to movie rating matrix like the Netflix setup. This will be a huge sparse matrix which is difficult to process. Note that each user will have a specific ...
Sanjay's user avatar
  • 151
5 votes

Recommender system based on purchase history, not ratings

there are no product ratings available, thus collaborative filtering is not an option Wrong. You can do collaborative filtering with holdings. Just use the numbers/duration of holdings instead of ...
Robin's user avatar
  • 1,337
5 votes

Calculate similarity on boolean data

You should look at the Jaccard Index, is the de facto similarity between set of items, where the sets are represented using a boolean vector. In this boolean vector each coordinate represents an item, ...
Dani Mesejo's user avatar
  • 2,226
5 votes

Mean Average Precision python code

This library called Metrics provides most of metrics for Machine Learning including MAP for Recommendation systems. If you only interested in metrics for recommendation systems, perhaps you can see ...
TwinPenguins's user avatar
  • 4,279
5 votes

How to use ndcg metric for binary relevance

The nDCG depends on the relevance of each document as you can see on the Wikipedia definition. I guess you could use 0 and 1 as relevance scores, but then all relevant documents would have the same ...
ovpira's user avatar
  • 51
4 votes

How can conclusions be drawn from recommendation systems evaluation?

"Good", I think, is based on the state of the art at the moment. So I would look at respected models from industry leaders and use their reported accuracies as a base line for what is "good": since ...
grldsndrs's user avatar
  • 567
4 votes

how to evaluate top n recommendation system with movie lens dataset?

For various metrics feel free to look at various benchmarking libraries including MyMediaLite and LibRec. If you are doing a TOP N approach, then the way to evaluate this using a Movielens system is ...
Skylion's user avatar
  • 141
4 votes
Accepted

Interpretation of an SVD for recommender systems

First of all, note that the dot product between two movies/users is by definition the correlation between them. So your intuition for treating it as a similarity measure is not wrong. Now, applying ...
ehudk's user avatar
  • 241
4 votes

Is there an overview over recommender system architectures?

A great source is the personal page of Xavier Amatriain (former Head of Engineering at Netflix and Quora). There are several links at publications detailing the system and architecture of recommender ...
Dani Mesejo's user avatar
  • 2,226
4 votes

Mean Average Precision python code

You can use the ml_metrics library. For install this library use: pip install ml_metrics import ml_metrics ml_metrics.mapk(actual, predicted, k)
yasin amini's user avatar
4 votes

Distance between users

No need for algorithms, or recommendation systems. You have: For each user a have a bunch of features. As long as they're numeric, or can be made numeric (e.g. aggregating the values or one-hot-...
grochmal's user avatar
  • 482
4 votes

What is difference between Nearest Neighbor and KNN?

Scikit wrote in his documantation: sklearn.neighbors provides functionality for unsupervised and supervised neighbors-based learning methods. Unsupervised nearest neighbors is the foundation of many ...
manpreet singh's user avatar
3 votes

How to use ML to forecast sales of a brand new product

You could use Bass or Gamma/Shifted Gompertz method to do that. You could use diffusion package in R. Here an example (it is ...
Econ_matrix's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible