I've a practical question in the areas of clustering/semantic search and would like to get some thoughts. Refer the figure for more details on this hypothetical situation.

Imagine I've 2 query embeddings (genre 1 and genre 2) and a bunch of movies I've embedded. Now, I need to limit my final list of movies that I recommend (assume the final list is limited to 20 recommendations. Also assume that each group in the figure has 10 movies for this situation).

So, I can do it in 2 ways:

  1. Global view - Look at all the distances together and give the 20 closest movies to any of the genre (so the final list of movies will be group 1, 2)
  2. Local view - Look at each query separately and give 10 closest movies (so the final list of movies will be group 1, 3)

Now, here's my thought process:

Global view: As I am giving the closest movies in global fashion, I reduce the False positives in my system. But I lose on the recall i.e I end up not having any movie recommended for genre 2.

Local view: As I am giving the closest movies in local fashion, I end up having a more representative recommendation system covering all genres. But I can end up in False positives (all movies in group 3 might not be relevant to genre 2; that's why high distance) and am missing out to recommend closer movies (group 2)

So, what's the best way to model the threshold in situations like this? Any high-level ideas, thoughts, references to academic publications, best practices are most welcome. I can share more details on the pipeline and methods I tried if it's helpful.

enter image description here

Note: The distances mentioned in the figure is for representation i.e it's not cluster distance or average distance and there're no groups! It's just for a clear explanation and to avoid cluttering. I've distances from query to every individual movie.

Real-world situation: I am dealing with ~30 million data points (each of 384-d) and have 10 queries. Assume for every query I've to return 100k closest items with least false-positives. So, any approach I follow has to be efficient (it's not time-sensitive)

Current idea: 30million embeddings --> FAISS (to limit to 1million closest points) --> t-SNE/UMAP (reduce 384-d to 50-d) --> HDBSCAN (to find the nearest cluster for a given query)



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.