其实很早就想知道如何将爬取到的数据存入数据库，并且实现前后台的交互功能，昨天刚刚看了一集关于爬数据并存数据的视频，今天，在这里总结一下~
以下为最终所需要爬取的信息：
由于需要爬取所有的二手商品信息，所以以下内容也要爬取到：
1.先写一个py文件，用于爬取上述图片类目导航的各个链接：
#-*-coding:utf-8-*- from bs4 import BeautifulSoup import requests start_url = 'http://bj.58.com/sale.shtml' url_host = 'http://bj.58.com' def get_channel_urls(url): wb_data = requests.get(start_url) soup = BeautifulSoup(wb_data.text,'html.parser') links = soup.select('ul.ym-submnu > li > b > a') for link in links: page_url = url_host + link.get('href') print page_url get_channel_urls(start_url) #类目导航链接 channel_list = ''' http://bj.58.com/shouji/ http://bj.58.com/tongxunyw/ http://bj.58.com/danche/ http://bj.58.com/fzixingche/ http://bj.58.com/diandongche/ http://bj.58.com/sanlunche/ http://bj.58.com/peijianzhuangbei/ http://bj.58.com/diannao/ http://bj.58.com/bijiben/ http://bj.58.com/pbdn/ http://bj.58.com/diannaopeijian/ http://bj.58.com/zhoubianshebei/ http://bj.58.com/shuma/ http://bj.58.com/shumaxiangji/ http://bj.58.com/mpsanmpsi/ http://bj.58.com/youxiji/ http://bj.58.com/jiadian/ http://bj.58.com/dianshiji/ http://bj.58.com/ershoukongtiao/ http://bj.58.com/xiyiji/ http://bj.58.com/bingxiang/ http://bj.58.com/binggui/ http://bj.58.com/chuang/ http://bj.58.com/ershoujiaju/ http://bj.58.com/bangongshebei/ http://bj.58.com/diannaohaocai/ http://bj.58.com/bangongjiaju/ http://bj.58.com/ershoushebei/ http://bj.58.com/yingyou/ http://bj.58.com/yingeryongpin/ http://bj.58.com/muyingweiyang/ http://bj.58.com/muyingtongchuang/ http://bj.58.com/yunfuyongpin/ http://bj.58.com/fushi/ http://bj.58.com/nanzhuang/ http://bj.58.com/fsxiemao/ http://bj.58.com/xiangbao/ http://bj.58.com/meirong/ http://bj.58.com/yishu/ http://bj.58.com/shufahuihua/ http://bj.58.com/zhubaoshipin/ http://bj.58.com/yuqi/ http://bj.58.com/tushu/ http://bj.58.com/tushubook/ http://bj.58.com/wenti/ http://bj.58.com/yundongfushi/ http://bj.58.com/jianshenqixie/ http://bj.58.com/huju/ http://bj.58.com/qiulei/ http://bj.58.com/yueqi/ http://bj.58.com/chengren/ http://bj.58.com/nvyongpin/ http://bj.58.com/qinglvqingqu/ http://bj.58.com/qingquneiyi/ http://bj.58.com/chengren/ http://bj.58.com/xiaoyuan/ http://bj.58.com/ershouqiugou/ http://bj.58.com/tiaozao/ http://bj.58.com/tiaozao/ http://bj.58.com/tiaozao/ '''
2.建立另一个py文件，用于爬取商品信息：
from bs4 import BeautifulSoup import requests import time import pymongo client = pymongo.MongoClient() ceshi = client['ceshi'] url_list = ceshi['url_list3'] item_info = ceshi['item_info3'] def get_links_from(channel,pages,who_sells=0): list_view = '{}{}/pn{}'.format(channel,str(who_sells),str(pages)) wb_data = requests.get(list_view) time.sleep(1) soup = BeautifulSoup(wb_data.text,'html.parser') if soup.find('td','t'): for link in soup.select('td.t a.t'): item_link = link.get('href').split('?')[0] url_list.insert_one({'url':item_link}) print item_link else: pass def get_item_info(url): wb_data = requests.get(url) soup = BeautifulSoup(wb_data.text,'html.parser') no_longer_exist = '404' in soup.find('script',type="text/javascript").get('src').split('/') if no_longer_exist: pass else: title = soup.title.text price = soup.select('span.price.c_f50')[0].text date = soup.select('.time')[0].text area = list(soup.select('.c_25d a')[0].stripped_strings) if soup.find_all('span','c_25d') else None item_info.insert_one({'title':title,'price':price,'date':date,'area':area}) print({'title':title,'price':price,'date':date,'area':area}) get_item_info('http://bj.58.com/shuma/28049255291945x.shtml') #get_links_from('http://bj.58.com/shuma/',2)
在第二个py文件中，需要注意的是：
1.连接mongodb时，可以不用提前在mongovue中建立数据库和collection，它会自动创建。
2.定义的第一个函数是用来爬取每个商品的具体url，并将其存入mongodb，第二个函数是通过每个商品具体的url来爬取相应的商品信息，并存入数据库，这两个函数不能同时运行。
3.在这次的爬虫项目中，有好多爬取技巧，如：
links = soup.select('ul.ym-submnu > li > b > a') list_view = '{}{}/pn{}'.format(channel,str(who_sells),str(pages)) for link in soup.select('td.t a.t'): item_link = link.get('href').split('?')[0] url_list.insert_one({'url':item_link}) area = list(soup.select('.c_25d a')[0].stripped_strings) if soup.find_all('span','c_25d') else None
等等，嘻嘻，反正这些我感觉比较新颖~~
最后，将数据存入数据库其实就是在连接数据库时，代码中有插入语句，就行~~
展示一下数据库中结果：
