If I have a very long list of paper names, how could I get abstract of these papers from internet or any database?

The paper names are like "Assessment of Utility in Web Mining for the Domain of Public Health".

Does any one know any API that can give me a solution? I tried to crawl google scholar, however, google blocked my crawler.

  • 2
    $\begingroup$ I doubt there's any general API for this. You can try crawling various services likes Academia.edu, publishers' sites and so on. Nevertheless, it would be easier to build a local database of documents first, and then experiment with extracting the abstracts. $\endgroup$ – Wojciech Walczak May 17 '14 at 8:55
  • $\begingroup$ Thanks for your answer! I have already built a local database for this. The problem of crawling from various services is that I have to make parse rules for each website. $\endgroup$ – Alex Gao May 17 '14 at 9:05
  • $\begingroup$ So, how about converting PDFs to TXTs and then extracting the abstracts with regular expressions? $\endgroup$ – Wojciech Walczak May 17 '14 at 9:35
  • $\begingroup$ thx! However, the contract states that massive download of papers are not allowed. This creates some headache. $\endgroup$ – Alex Gao May 17 '14 at 11:39
  • 2
    $\begingroup$ I think this stack-overflow answer link gives the best answer I can get. Maybe people who encounter this problem could also have a look at this page. $\endgroup$ – Alex Gao May 17 '14 at 11:55

Look it up on:

If you get a single exact title match then you have probably found the right article, and can fill in the rest of the info from there. Both give you download links and bibtex-style output. What you would likely want to do though to get perfect metadata is download and parse the pdf (if any) and look for DOI-style identifier.

Please be nice and rate-limit your requests if you do this.


arXiv has an API and bulk download but if you want something for paid journals it will be hard to come by without paying an indexer like pubmed or elsevier or the like.

  • 1
    $\begingroup$ thanks a lot. However arXiv does provide the papers I need. $\endgroup$ – Alex Gao May 19 '14 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.