3
$\begingroup$

We have ~500 biomedical documents each of some 1-2 MB. We want to use a non query-based method to rank the documents in order of their unique content score. I'm calling it "unique content" because our researchers want to know from which document to start reading. All the documents are of the same topic, in the biomedical world we know that there is always a lot of content overlap. So all we want to do is to arrange the documents in the order of their unique content.

Most Information Retrieval literature suggest query-based ranking which does not fit our need.

Improve this question
$\endgroup$
2
  • $\begingroup$ From this description, I'm not at all clear what the ranking is supposed to be. There is no such thing as ordering by "unique content". Clarify please what the ordering is supposed to be determined by? $\endgroup$
    – Sean Owen
    Commented Sep 8, 2014 at 9:15
  • $\begingroup$ @SeanOwen All documents come under the same theme for instance Tissue engineering for Bonemarrow related cancer. Within this theme we know that most research article will definetly speak about genes (ABC) so this becomes the predominant theme within those topics. Most of these articles dwell within these same set of genes. But very few articles start speak about latent themes which involves genes (ADF) and (XYZ). The current thought process is we want to rank the articles based on the content (XYZ),(FDA),(ABC). This is what I meant by unique content. $\endgroup$
    – DACW
    Commented Sep 10, 2014 at 10:49

2 Answers 2

Reset to default
2
$\begingroup$

You could use Topic Modeling as described in this paper: http://faculty.chicagobooth.edu/workshops/orgs-markets/pdf/KaplanSwordWin2014.pdf

They performed Topic Modeling on abstracts of patents (limited to 150 words). They identified papers as "novel" if they were the first to introduce a topic, and measured degree of novelty by how many papers in the following year used the same topic. (Read the paper for details).

I suggest that you follow their lead and only process paper abstracts. Processing the body of each paper might reveal some novelty that the abstract does not, but you also run the risk of having much more noise in your topic model (i.e. extraneous topics, extraneous words).

While you say that all 500 papers are on the same "topic", it's probably safer to say that they are all on the same "theme" or in the same "sub-category" of Bio-medicine. Topic modeling permits decomposition of the "theme" into "topics".

The good news is that there are plenty of good packages/libraries for Topic Modeling. You still have to do preprocessing, but you don't have to code the algorithms yourself. See this page for many resources: http://www.cs.princeton.edu/~blei/topicmodeling.html

Improve this answer
$\endgroup$
0
1
$\begingroup$

Here's a simple initial approach to try:

  1. Calculate the TF-IDF score of each word in each document.
  2. Sort the documents by the average TF-IDF score of their words.
  3. The higher the average TF-IDF score, the more unique a document is with respect to the rest of the collection.

You might also try a clustering-based approach where you look for outliers, or perhaps something with the Jaccard index using a bag-of-words model.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks. Im looking at weighted TF-IDF in cases were the key-words are already known . any suggestions for that ? $\endgroup$
    – DACW
    Commented Sep 10, 2014 at 11:00
  • $\begingroup$ What are you hoping to achieve by boosting the TF-IDF scores of known keywords? $\endgroup$
    – Charlie Greenbacker
    Commented Sep 11, 2014 at 13:58
  • $\begingroup$ Exploring weighted TD-IDF to introduce conscious bias of sorts. $\endgroup$
    – DACW
    Commented Sep 17, 2014 at 13:30
  • $\begingroup$ If you have a controlled vocabulary of terms you care about, you could boost the score of those specific items. $\endgroup$
    – Charlie Greenbacker
    Commented Sep 18, 2014 at 13:40
  • $\begingroup$ A quick update .. Ive implemented this logic and the results have come out pretty well. Built on Python NLTK and gensim. im exploring opportunities to use information gain to add in some measure of "usefulness" using information gain or something like that . LSI is also an option here. $\endgroup$
    – DACW
    Commented Oct 9, 2014 at 21:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.