15
$\begingroup$

If I have a very long list of paper names, how could I get abstract of these papers from internet or any database?

The paper names are like "Assessment of Utility in Web Mining for the Domain of Public Health".

Does any one know any API that can give me a solution? I tried to crawl google scholar, however, google blocked my crawler.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ I doubt there's any general API for this. You can try crawling various services likes Academia.edu, publishers' sites and so on. Nevertheless, it would be easier to build a local database of documents first, and then experiment with extracting the abstracts. $\endgroup$
    – Wojciech Walczak
    Commented May 17, 2014 at 8:55
  • $\begingroup$ Thanks for your answer! I have already built a local database for this. The problem of crawling from various services is that I have to make parse rules for each website. $\endgroup$
    – Alex Gao
    Commented May 17, 2014 at 9:05
  • $\begingroup$ So, how about converting PDFs to TXTs and then extracting the abstracts with regular expressions? $\endgroup$
    – Wojciech Walczak
    Commented May 17, 2014 at 9:35
  • $\begingroup$ thx! However, the contract states that massive download of papers are not allowed. This creates some headache. $\endgroup$
    – Alex Gao
    Commented May 17, 2014 at 11:39
  • 2
    $\begingroup$ I think this stack-overflow answer link gives the best answer I can get. Maybe people who encounter this problem could also have a look at this page. $\endgroup$
    – Alex Gao
    Commented May 17, 2014 at 11:55

2 Answers 2

Reset to default
9
$\begingroup$

Look it up on:

If you get a single exact title match then you have probably found the right article, and can fill in the rest of the info from there. Both give you download links and bibtex-style output. What you would likely want to do though to get perfect metadata is download and parse the pdf (if any) and look for DOI-style identifier.

Please be nice and rate-limit your requests if you do this.

Improve this answer
$\endgroup$
0
5
$\begingroup$

arXiv has an API and bulk download but if you want something for paid journals it will be hard to come by without paying an indexer like pubmed or elsevier or the like.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ thanks a lot. However arXiv does provide the papers I need. $\endgroup$
    – Alex Gao
    Commented May 19, 2014 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.