I am trying to build a scrapy bot capable of ripping the data from my local craigslist for jobs as well as having a recursive functionality to allow for the contact data to be gathered as well. Ultimately I would like to have all of this data placed into a .CVS file. I have read Learning Python the Hard Way as well as a good portion of Automating the hard stuff with Python, however I am still quite a novice. I have referenced the following tutorials while trying to build this script: the documentation for Scrapy & this blog post by Michael Herman. I feel as though my code simply has a dumb syntax error but I'm too ignorant of what to do to fix it.
Here's the code for the spider:
from scrapy.contrib.spiders import CrawlSpider, Rule from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor from scrapy.selector import HtmlXPathSelector from craigslist_sample.items import CraigslistSampleItem class MySpider(CrawlSpider): name = "craigs" allowed_domains = ["sfbay.craigslist.org"] start_urls = ["http://salem.craigslist.org/search/jjj"] #Initially grab all of the urls up to where craigslist allows #In this case, it's 2400 for i in range(1, 24): start_urls.append(base_url + "s=" + str(i) + "00&") rules = ( Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True), ) def parse_items(self, response): hxs = HtmlXPathSelector(response) titles = hxs.xpath('//span[@class="pl"]') items =  for titles in titles: item = CraigslistSampleItem() item["title"] = titles.xpath("a/text()").extract() item["link"] = titles.xpath("a/@href").extract() item["email"] = titles.xpath("@anonemail").extract() items.append(item) return(items)
The following is the exact error I am receiving:
*File "C:\Users\newfa\Documents\scripts\craigslist_sample\craigslist_sample\spiders\test.py", line 27 item["email"] = titles.xpath("@anonemail").extract() ^
IndentationError: unexpected indent*
I do understand that indentation is very important in Python, however I keep trying different methods of indentation and have tried several "beautify" code methods to try and get it right, which leads me to believe that it may be some other error.