# Trouble with Scrapy and Python: ever present unexpected indent

I am trying to build a scrapy bot capable of ripping the data from my local craigslist for jobs as well as having a recursive functionality to allow for the contact data to be gathered as well. Ultimately I would like to have all of this data placed into a .CVS file. I have read Learning Python the Hard Way as well as a good portion of Automating the hard stuff with Python, however I am still quite a novice. I have referenced the following tutorials while trying to build this script: the documentation for Scrapy & this blog post by Michael Herman. I feel as though my code simply has a dumb syntax error but I'm too ignorant of what to do to fix it.

Here's the code for the spider:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.selector import HtmlXPathSelector
from craigslist_sample.items import CraigslistSampleItem

class MySpider(CrawlSpider):
name = "craigs"
allowed_domains = ["sfbay.craigslist.org"]
start_urls = ["http://salem.craigslist.org/search/jjj"]
#Initially grab all of the urls up to where craigslist allows
#In this case, it's 2400
for i in range(1, 24):
start_urls.append(base_url + "s=" + str(i) + "00&")

rules = (
Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True),
)

def parse_items(self, response):
hxs = HtmlXPathSelector(response)
titles = hxs.xpath('//span[@class="pl"]')
items = []
for titles in titles:
item = CraigslistSampleItem()
item["title"] = titles.xpath("a/text()").extract()
item["email"] = titles.xpath("@anonemail").extract()
items.append(item)
return(items)


The following is the exact error I am receiving:

*File "C:\Users\newfa\Documents\scripts\craigslist_sample\craigslist_sample\spiders\test.py", line 27
item["email"] = titles.xpath("@anonemail").extract()
^


IndentationError: unexpected indent*

I do understand that indentation is very important in Python, however I keep trying different methods of indentation and have tried several "beautify" code methods to try and get it right, which leads me to believe that it may be some other error.

After moments more of work, I was capable of solving my own problem. The error turned out to be a wild 'tab' mark that I had lost in the code there. Also, my 'allowed_domains' section was wonky. This is all corrected in the code below

 from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.selector import HtmlXPathSelector
from craigslist_sample.items import CraigslistSampleItem

class MySpider(CrawlSpider):
name = "test"
allowed_domains = ["salem.craigslist.org"]
start_urls = ["http://salem.craigslist.org/search/jjj"]

rules = (
Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True),
)

def parse_items(self, response):
hxs = HtmlXPathSelector(response)
titles = hxs.xpath('//span[@class="pl"]')
items = []
for titles in titles:
item = CraigslistSampleItem()
item["title"] = titles.xpath("a/text()").extract()