1
$\begingroup$

Given an index or database with a lot of (short) documents (~ 1 million), I am trying to do some kind of novelty detection for each newly incoming document.

I know that I have to compute the similarity of the new document with each document in the index. If the similarity is below a certain threshold, one can consider this document as novel. One common approach - which I want to do - is to use a Vector Space Model and compute the cosine similarity (e.g. by using Apache Lucene).

But this approach has two shortcomings: 1) it is computationally expensive and 2) it does not incorporate the semantics of documents and words respectively.

In order to overcome these shortcomings, my idea was to either use an LDA topic distribution or named entities to augment the Lucene index and the query (i.e. the document collection and each new document) with semantics.

Now, I am completely lost regarding the concrete implementation. I have already trained an LDA topic model using Mallet and I am also able to do Named Entity Recognition on the corpus. But I do not know how to use these topics and named entities in order to realise novelty detection. More specifically, I do not know how to use these features for index and query creation.

For example, is it already sufficient to store all named entities of one document as a separate field in the index, add certain weights (i.e. boost them) and use a MultiFieldQuery? I do not think that this already adds some kind of semantics to the similarity detection. The same applies to LDA topics: is it sufficient to add the topic probability of each term as a Payload and implement a new similarity score?

I would be very happy if you could provide some hints or even code snippets on how to incorporate LDA topics or named entities in Lucene for some kind of novelty detection or semantic similarity measure.

Thank you in advance.

Improve this question
$\endgroup$
1
  • $\begingroup$ Use a pretrained document embedding model (e.g., doc2vec), and approach it is as an online clustering problem. Documents that do not belong to an existing cluster are novel. The cosine product is your measure of similarity. $\endgroup$
    – Emre
    Mar 24, 2017 at 17:05

1 Answer 1

Reset to default
1
$\begingroup$

Short answer: Topic Models and LDA aren't suited at all for novelty detection. I did some experiments with data that show a succession of topics in temporal order, and at the boundaries of the time interval one topic gets really high percentages in the document-topic matrix, i.e., the upcoming new trend (or the outrunning old trend) is not seen by LDA.

Topic Models are also not really suited for "online" processing—you read the whole corpus of documents once and than infer the topics.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for this insight. But what about incremental topic models like online LDA? Would they be more suitable? $\endgroup$
    – phly
    Mar 24, 2017 at 15:26
  • $\begingroup$ Unfortunately, the link to the original paper in the link you quote is broken. In addition, I cannot tell you about online LDA, because I have not done it yet (but I am eager to learn about it). $\endgroup$
    – user10169
    Mar 24, 2017 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.