爬虫参考资料

开源爬虫软件汇总：http://blog.chinaunix.net/uid-22414998-id-3774291.html

Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
强大的scrapy爬虫框架（Python）：http://scrapy.org/
Python抓取框架：Scrapy的架构： http://www.biaodianfu.com/scrapy-architecture.html
使用scrapy进行大规模抓取：http://www.yakergong.net/blog/archives/500
Scrapy入门教程：http://www.cnblogs.com/txw1958/archive/2012/07/16/scrapy-tutorial.html
一个scrapy例子：https://github.com/scrapy/dirbot
一个分布式定向抓取集群的简单实现：https://github.com/agathewiky/spider-roach