scrapy

前段时间项目使用了scrapy,这里做个简单的demo,使用scrapy抓取下安居客的内容, 关于怎么搭建scrapy的工程和scrapy部分概念的使用，请查看https://blog.csdn.net/mingover/article/details/80717974

全部源代码:

https://github.com/huawumingguo/scrapy_demo

分析安居客页面

是否要登陆?

我们发现安居客查看是不用登陆的，我们把内容搞简单些，只抓取一个城市的内容，如广州: https://guangzhou.anjuke.com/sale/rd1/?kw=&from=sugg

上面是一个房源列表，里面有下一页，我们对第一页进行抓取，然后把详情 url,和下一页url都加到抓取目标中来.

简略的流程:

def parse(): 获取详情列表items for items: yield Request(url=parse.urljoin(response.url, itemurl[0]), meta={"refer_url": response.url}, callback=self.parse_detail, headers=self.headers) 获取下一页的链接nexturl yield Request(url=parse.urljoin(response.url, nexturl), callback=self.parse, headers=self.headers) def parse_detail(): 解析列表后,扔给pipeline item = citemloader.load_item() yield item

pipeline拿到相关内容后，写入mysql/其它持久化

处理安居客相应内容

直接代码：

# -*- coding: utf-8 -*- import scrapy from scrapy.http import Request from urllib import parse from scrapy.loader import ItemLoader from anjuke.items import * import re import logging from anjuke.utils.LoggerUtil import LogUtils loggerDataName = "anjuke" log_dataInfo_path = "logs/anjuke.log" log = LogUtils.createLogger(loggerDataName, log_dataInfo_path) class AnjukeGzSpider(scrapy.Spider): name = 'anjuke_gz' allowed_domains = ['anjuke.com'] start_urls = ['https://guangzhou.anjuke.com/sale/'] headers = { "HOST": "guangzhou.anjuke.com", "Referer": "https://guangzhou.anjuke.com", 'User-Agent': "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0" } def parse(self, response): items = response.css("#houselist-mod-new > li") log.info("正在处理_list:%s,条数:%s" %( response.url ,len(items))) # 获取列表 for itemnode in items: itemurl = itemnode.css('a.houseListTitle::attr(href)').extract() itemtitle = itemnode.css('a.houseListTitle::attr(title)').extract() log.info("list中显示:%s,标题:%s" % (itemurl, itemtitle)) # print('%s,链接为:%s' % (itemtitle[0], itemurl[0])) yield Request(url=parse.urljoin(response.url, itemurl[0]), meta={"refer_url": response.url}, callback=self.parse_detail, headers=self.headers) # 获取下一页 nexturlArr = response.css('#content > div.sale-left > div.multi-page > a.aNxt::attr(href)').extract() if nexturlArr: nexturl = nexturlArr[0] yield Request(url=parse.urljoin(response.url, nexturl), callback=self.parse, headers=self.headers) def parse_detail(self, response): # "https://guangzhou.anjuke.com/prop/view/A1285389340?from=filter&spread=commsearch_p&position=361&kwtype=filter&now_time=1529215280" refer_url = response.meta.get("refer_url", '') log.info("正在处理_detail:%s,refer_url为：%s" %( response.url,refer_url)) match_re = re.match(".*view/(.*)\?.*", response.url) id = match_re.group(1) citemloader = OrderItemLoader(item=OrderItem(), response=response) citemloader.add_value("id", id) citemloader.add_css("title", "#content > div.clearfix.title-guarantee > h3::text") citemloader.add_css("community_id", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.first-col.detail-col > dl:nth-child(1) > dd > a::attr(href)") citemloader.add_css("community_name", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.first-col.detail-col > dl:nth-child(1) > dd > a::text") citemloader.add_css("area1", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.first-col.detail-col > dl:nth-child(2) > dd > p > a:nth-child(1)::text") citemloader.add_css("area2", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.first-col.detail-col > dl:nth-child(2) > dd > p > a:nth-child(2)::text") citemloader.add_css("build_time", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.first-col.detail-col > dl:nth-child(3) > dd::text") citemloader.add_css("address", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.first-col.detail-col > dl:nth-child(2) > dd > p::text") citemloader.add_css("housetype", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.first-col.detail-col > dl:nth-child(4) > dd::text") citemloader.add_css("housestructure", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.second-col.detail-col > dl:nth-child(1) > dd::text") citemloader.add_css("space", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.second-col.detail-col > dl:nth-child(2) > dd::text") citemloader.add_css("building_floors", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.second-col.detail-col > dl:nth-child(4) > dd::text") citemloader.add_css("house_floor", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.second-col.detail-col > dl:nth-child(4) > dd::text") citemloader.add_css("direction_face", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.second-col.detail-col > dl:nth-child(3) > dd::text") citemloader.add_css("unit_price", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.third-col.detail-col > dl:nth-child(1) > dd::text") citemloader.add_css("consult_first_pay", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.third-col.detail-col > dl:nth-child(2) > dd::text") citemloader.add_css("decoration_degree", "#content > div.wrapper > div.wrapper-lf.clearfix > div.houseInfoBox > div > div.houseInfo-wrap > div > div.third-col.detail-col > dl:nth-child(4) > dd::text") import datetime nowTime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S') citemloader.add_value("refer_url", refer_url) citemloader.add_value("now_time", nowTime) item = citemloader.load_item() yield item pass

这里花时间最多的是解析各种css,主要是在item中的正则表达式中处理

把item给定义好

def re_community_id(value): match_re = re.match(".*view/(\d+)$", value) result = match_re.group(1) return result def re_address(value): try: result = re.match(".*(\s+)(.*)", value).group(2) except Exception as e: print(e) result = 'aaaa' return result def re_build_time(value): return re.match("(.*)年", value).group(1) def re_space(value): return re.match("(.*)平方米", value).group(1) def re_building_floors(value): return re.match(".*共(\d+)层", value).group(1) def re_house_floor(value): # 一般是 高层(共10层) 这样，也有 共5层 target = re.match("(.*)\(", value) if target: return target.group(1) else: return value.strip() def re_unit_price(value): return re.match("(.*)元.*", value).group(1) def get_second(arr): if len(arr) > 0: if len(arr) > 1: return arr[1] else: return arr[0] else: return "cccc" def re_consult_first_pay(value): try: target = value.strip() result = re.match("(.*)万", target).group(1) except Exception as e: print(e) result = 0 return int(float(result) * 10000) def re_consult_month_pay(value): return re.match("(.*)元.*", value).group(1) def getval(value): return value def getridof_blank(value): if value: valstr = value[0] target = ''.join(valstr.split()) return target class OrderItemLoader(ItemLoader): # 自定义itemloader default_output_processor = TakeFirst() class OrderItem(scrapy.Item): id = scrapy.Field() title = scrapy.Field( output_processor=getridof_blank ) community_id = scrapy.Field( input_processor=MapCompose(re_community_id), ) community_name = scrapy.Field() area1 = scrapy.Field() area2 = scrapy.Field() address = scrapy.Field( input_processor=MapCompose(re_address), output_processor=get_second ) build_time = scrapy.Field( input_processor=MapCompose(re_build_time), ) housetype = scrapy.Field() housestructure = scrapy.Field( output_processor=getridof_blank ) space = scrapy.Field( input_processor=MapCompose(re_space), ) building_floors = scrapy.Field( input_processor=MapCompose(re_building_floors), ) house_floor = scrapy.Field( input_processor=MapCompose(re_house_floor) ) direction_face = scrapy.Field() unit_price = scrapy.Field( input_processor=MapCompose(re_unit_price) ) consult_first_pay = scrapy.Field( input_processor=MapCompose(re_consult_first_pay) ) decoration_degree = scrapy.Field() hosedesc = scrapy.Field() refer_url = scrapy.Field() now_time = scrapy.Field()

这里itemloader就是解析html的内容，要有耐心慢慢调，这里就是最花时间的地方

扔到mysql中

持久化可以有多种方式了, json或mysql之类的都行,这里提供几种:

class JsonWithEncodingPipeline(object): #自定义json文件的导出 def __init__(self): self.file = codecs.open('anjuke_orderlist.json', 'w', encoding="utf-8") def process_item(self, item, spider): lines = json.dumps(dict(item), ensure_ascii=False) + "\n" self.file.write(lines) self.file.flush() return item def spider_closed(self, spider): self.file.close() class MysqlPipeline(object): #采用同步的机制写入mysql def __init__(self): self.conn = MySQLdb.connect('192.168.0.106', 'root', 'root', 'article_spider', charset="utf8", use_unicode=True) self.cursor = self.conn.cursor() def process_item(self, item, spider): insert_sql = """ insert into jobbole_article(title, url, create_date, fav_nums) VALUES (%s, %s, %s, %s) """ self.cursor.execute(insert_sql, (item["title"], item["url"], item["create_date"], item["fav_nums"])) self.conn.commit() class MysqlTwistedPipline(object): def __init__(self, dbpool): self.dbpool = dbpool @classmethod def from_settings(cls, settings): dbparms = dict( host = settings["MYSQL_HOST"], db = settings["MYSQL_DBNAME"], user = settings["MYSQL_USER"], passwd = settings["MYSQL_PASSWORD"], charset='utf8', cursorclass=MySQLdb.cursors.DictCursor, # use_unicode=True, ) dbpool = adbapi.ConnectionPool("MySQLdb", **dbparms) return cls(dbpool) def process_item(self, item, spider): #使用twisted将mysql插入变成异步执行 query = self.dbpool.runInteraction(self.do_insert, item) query.addErrback(self.handle_error, item, spider) #处理异常 def handle_error(self, failure, item, spider): #处理异步插入的异常 print (failure) def do_insert(self, cursor, item): #执行具体的插入 #根据不同的item 构建不同的sql语句并插入到mysql中 insert_sql, params = item.get_insert_sql() # targetparams = [] # try: # for param in params: # if isinstance(param, str): # targetparam = param.encode('utf8').decode('utf8') # targetparams.append(targetparam) # else: # targetparams.append(targetparam) # except Exception as e: # print (e) # raise e insert_sql = insert_sql.strip() try: cursor.execute(insert_sql, params) except Exception as e: print (e) pass

怎么保证去重,不会是原来的数据

一些说明

怎么迁移workon 的目录?

workon 默认的目录是C:\Users\xxxx\Envs 把WROKON_HOME的路径自定义到指定目录即可 之前已经定义好的workon 则在老目录中直接COPY过去即可

shell 要加上user_agent

参考：　http://www.bubuko.com/infodetail-2145085.html

scrapy shell -s USER_AGENT=”Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.86 Safari/537.36” https://guangzhou.anjuke.com/prop/view/A1275284529?from=filter&spread=commsearch_p&position=1&kwtype=filter&now_time=1529290160

我们用第三方工具来搞，的downloadmiddleware

class RandomUserAgentMiddlware(object): #随机更换user-agent def __init__(self, crawler): super(RandomUserAgentMiddlware, self).__init__() self.ua = UserAgent() self.ua_type = crawler.settings.get("RANDOM_UA_TYPE", "random") @classmethod def from_crawler(cls, crawler): return cls(crawler) def process_request(self, request, spider): def get_ua(): return getattr(self.ua, self.ua_type) tmpua = get_ua() request.headers.setdefault('User-Agent', tmpua)

怎么用代理？

实际上，代理就是request.meta[“proxy”] = “xxxx” 加上就行了，不过一般网上的代理都是一段时间就失效的，我们需要构建一个代理池，来动态获取，遇到不行的，就略过. 参考代码中的GetProxy.py

怎么抓取html内容

response.css 如果是里面想抓所有的html , ::text是只能获取文字的,html内容不会被抓取 那么，不加::text即可

运行pip报错：Fatal error in launcher: Unable to create process using ‘”’

https://blog.csdn.net/cjeric/article/details/73518782

关于字符串编码

https://www.cnblogs.com/geekard/archive/2012/10/04/python-string-endec.html

运行过程中存好这些jobs

scrapy crawl anjuke_gz -s JOBDIR=jobs/job1

如果要关闭，就ctrl + c

怎么把一些参数传给下一步数据

Request时候加上meta，传给callback中的 response对象 yield Request(url=parse.urljoin(response.url, itemurl[0]), meta={"refer_url": response.url}, callback=self.parse_detail, headers=self.headers) .... response中拿到: refer_url = response.meta.get("refer_url", '')

通过job 后，是否可以不去重复执行了?

是的，关掉重启后，指定同一个job,会在之前的基础上去抓取

后记

发现安居客一个奇怪的问题，找了几个城市，发现个个城市都是50页，感觉安居客的房源不太OK,这些代码只可以留来简单学习。 https://www.zhihu.com/question/20594581/answer/15585452

在linux的安装包情况

安装sqlite3 sqlite3有点兼容问题，不能通过pip搞定 https://blog.csdn.net/sparkexpert/article/details/69944835 https://github.com/sloria/TextBlob/issues/173

实际上就是安装下libsqlite3-dev,然后重新编译python3

1,sudo apt-get install libsqlite3-dev 2,(这一步这边没有执行)(Or you can install more packages as suggested on the pyenv wiki: apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev 3,Now in the downloaded python source rebuild and install python with the following command: ./configure --enable-loadable-sqlite-extensions && make && sudo make install

安装 Twisted,这货也有个bug,不能直接pip安装. https://pypi.org/project/Twisted/#files 这里下载后手动安装，安装过程百度下

pip install scrapy pip install fake_useragent pip install requests pip install kafka pip install redis pip install pillow apt-get install libmysqlclient-dev pip install mysqlclient

参考:

https://coding.imooc.com/class/92.html