1
$\begingroup$

I am trying to build a scrapy bot capable of ripping the data from my local craigslist for jobs as well as having a recursive functionality to allow for the contact data to be gathered as well. Ultimately I would like to have all of this data placed into a .CVS file. I have read Learning Python the Hard Way as well as a good portion of Automating the hard stuff with Python, however I am still quite a novice. I have referenced the following tutorials while trying to build this script: the documentation for Scrapy & this blog post by Michael Herman. I feel as though my code simply has a dumb syntax error but I'm too ignorant of what to do to fix it.

Here's the code for the spider:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from craigslist_sample.items import CraigslistSampleItem

class MySpider(CrawlSpider):
name = "craigs"
allowed_domains = ["sfbay.craigslist.org"]
start_urls = ["http://salem.craigslist.org/search/jjj"]
#Initially grab all of the urls up to where craigslist allows
#In this case, it's 2400 
for i in range(1, 24):
    start_urls.append(base_url + "s=" + str(i) + "00&")

rules = (
    Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True),
)

def parse_items(self, response):
    hxs = HtmlXPathSelector(response)
    titles = hxs.xpath('//span[@class="pl"]')
    items = []
    for titles in titles:
        item = CraigslistSampleItem()
        item["title"] = titles.xpath("a/text()").extract()
        item["link"] = titles.xpath("a/@href").extract()
        item["email"] = titles.xpath("@anonemail").extract()
        items.append(item)
    return(items)

The following is the exact error I am receiving:

*File "C:\Users\newfa\Documents\scripts\craigslist_sample\craigslist_sample\spiders\test.py", line 27
item["email"] = titles.xpath("@anonemail").extract()
^

IndentationError: unexpected indent*

I do understand that indentation is very important in Python, however I keep trying different methods of indentation and have tried several "beautify" code methods to try and get it right, which leads me to believe that it may be some other error.

$\endgroup$
1
$\begingroup$

After moments more of work, I was capable of solving my own problem. The error turned out to be a wild 'tab' mark that I had lost in the code there. Also, my 'allowed_domains' section was wonky. This is all corrected in the code below

 from scrapy.contrib.spiders import CrawlSpider, Rule
 from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
 from scrapy.selector import HtmlXPathSelector
 from craigslist_sample.items import CraigslistSampleItem

class MySpider(CrawlSpider):
name = "test"
allowed_domains = ["salem.craigslist.org"]
start_urls = ["http://salem.craigslist.org/search/jjj"]


rules = (
    Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True),
)

def parse_items(self, response):
    hxs = HtmlXPathSelector(response)
    titles = hxs.xpath('//span[@class="pl"]')
    items = []
    for titles in titles:
       item = CraigslistSampleItem()
       item["title"] = titles.xpath("a/text()").extract()
       item["link"] = titles.xpath("a/@href").extract()
       item["email"] = titles.xpath("A/@anonemail").extract()
       items.append(item)
    return(items)

Unfortunately, I am still not managing to acquire the email, phone number, or description data - however as these problems are not directly related to the question I posed its fair to say that it is answered.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Make sure to use a proper IDE that will take care of the indentation for you, that way a problem like this will never happen (or it will at least be visible where it is) $\endgroup$ – Jan van der Vegt Sep 2 '16 at 7:06
  • $\begingroup$ Thanks yeah, I like to use Notepad++ just because I have a cheapo laptop and having one light editor is easier - however I will be using an IDE here on out because of exactly the reason you stated. $\endgroup$ – Zach Ross Sep 2 '16 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.