Basically, the idea is to have users following tags on the site, so each users has a set of tags they are following. And then there is a document collection where each document in the collection has a Title, Description, and a set of tags which are of relevance to the topic being discussed in the document as determined by the author. What is the best way of recommending documents to the user, given the information we have, which would also take into consideration the semantic relevance of the title and description of a document to the user's tags, whether that be a word embeddings solution or a tf-idf solution, or a mix, do tell. I still don't know what I'm going to do about tag synonyms, it might have to be a collaborative effort like on stackoverflow, but if there is a solution to this or a pseudo-solution, and I'm writing this in C# using the Lucene.NET library, if that is of any relevance to you.


2 Answers 2


If I got your problem description correct, you are looking for a recommender system like for example used by Netflix or Amazon. State of the art solution would be to use Latent Dirichlet Allocation topic modeling to make recommendations based on topics (in your case, topics would be the tags). Here is a very good video tutorial on this topic: https://youtu.be/3mHy4OSyRf0

In the case of the standard version of LDA, you don't even have to define the tags, you just define a value of different tags among all your documents. If you have for example 10000 documents and you want to use 100 different tags, the method will transform your words/documents matrix into a topics/documents matrix.

The entries of the words/documents matrix are simply all documents as columns and all words (from all your documents) as rows, then for each document you have the counts of each word.

The entries of the topics/documents matrix are all documents as columns and all possible topics as rows, then for each document you have entries like 78% topic1, 12.5% topic95, 0% topic99 on each topic.

Once you have this data and you want to recommend a new document to a user based on his interests(tags), or in other words you have a user_interests vector $\vec{u}$ with 100 entries which have values between 0 and 1, and you have topics/documents matrix $M_{{topics}\times{documents}}$ you calculate a new matrix by multiplaying $M_{{topics}\times{documents}}*\vec{u}$, from this matrix you calculate the sum from each row and recommend those documents with the highest sum.

If you just want to use predefined tags, you can skip the step where you use the LDA method to calculate the topics/documents matrix and simply use your data to represent your documents as tags/documents matrix and your users as tag_vectors, proceeding from here the same as above: multiplying the matrix with a user_vector, calculating the sum from each row and recommending the documents with the highest sum.

  • $\begingroup$ One option as you say is to generate topics from the document collection, in which case each document consists of a Title and Description, then generating some user_interests vector based on likes and/or past history. This way also allows for the possibility of recommending users topics to follow much like pinterest. Also since LDA doesn’t give you back a topic but rather a group of words corresponding to a certain topic, would it be wise to train a word2vec model in order to get some average vector of all the words and call that the topic? also how does LDA deal with synonymous topics? $\endgroup$ Commented Mar 11, 2020 at 22:14
  • $\begingroup$ also what do you think about lda2vec? $\endgroup$ Commented Mar 12, 2020 at 18:29
  • $\begingroup$ There are many modified versions of LDA available as of today, I'm only familar with the base method as described in the David M. Blei, Andrew Y. Ng, Michael I. Jordan; 3(Jan):993-1022, 2003 paper here: web.archive.org/web/20120501152722/http://jmlr.csail.mit.edu/… I saw a paper once of a modified version which was able to use a data base for finding exact tags for each topic, depending on the words constellations of the words represented by the topic. $\endgroup$
    – Eugen
    Commented Mar 12, 2020 at 23:26
  • $\begingroup$ I really like LDA2Vec and the kind of statistics I can capture with it, but my only worry is having to retrain these word embeddings because of my corpus constantly changing(new chats being added), word embeddings seem to work well for more static corpuses i.e netflix movies or spotify songs, but I'm not sure how I would go about it in a situation like this. Any ideas? $\endgroup$ Commented Mar 25, 2020 at 15:25

One solution could be to train a single embedding space, StarSpace is one such implementation. That single embedding space would contain all users, documents, and tags. Then it is a nearest neighbor search to recommend any combination. Given a user, find the nearest documents. Given a tag, find the nearest tags …

For new entities (i.e., users, documents, or tags), split the individual entity into parts. For example, a document will have tokens or tag will be associated with documents that have tokens. Then find the average embedding of all the parts. That location in the embedding space is the approximately semantic meaning of new entities.

Overall, this is a complex, open-end problem, thus there are many possible ways to create useful solutions. The next best solutions depend on what has already been done and what the next feature that would add the most value to end-users.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.