Python爬虫 — Scrapy爬取IT桔子网

Posted on Updated on

转载自   成长之路丶关注@简书

目标：

此次爬取主要是针对IT桔子网的事件信息模块，然后把爬取的数据存储到mysql数据库中。

 

aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNjc1NTM5MC0xODdmZDlhZDdlM2Y0ZTdhLnBuZw

 

aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNjc1NTM5MC1mZWFhYjQ4ZTAwODQyYmZiLnBuZw

目标分析：

通过浏览器浏览发现事件模块需要登录才能访问，因此我们需要先登录，抓取登录接口：

aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNjc1NTM5MC05OGI2M2QxZDdkZjg4ZTBiLnBuZw

可以看到桔子网的登录接口是：https://www.itjuzi.com/api/authorizations，请求方式是post请求，数据的提交方式是payload，提交的数据格式是json(payload方式提交的数据一般都是json)，再看一下响应：

aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNjc1NTM5MC1iZWQwYzI0NDVhMjIyY2RmLnBuZw

发现没有响应数据，其实是有响应数据的，只是F12调试看不到，我们可以用postman来查看响应体：

aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNjc1NTM5MC05NjE0YjM0MWFlZmRkMzI0LnBuZw

可以发现响应体是json数据，我们先把它放到一边，我们再来分析事件模块，通过F12抓包调试发现事件模块的数据其实是一个ajax请求得到的：

aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNjc1NTM5MC1kYTY2Y2MwZjc4OWIyZTRmLnBuZw

ajax请求得到的是json数据，我们再看看headers：

aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNjc1NTM5MC1lNTg3ZDJjZDQ3NmE3MDI3LnBuZw

 

aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNjc1NTM5MC1kZTUxMmQzNTk0MjBkZGIxLnBuZw

 

aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNjc1NTM5MC0zNDg1M2IwN2M0ODg0NjNhLnBuZw

可以发现headers里有一个Authorization参数，参数的值恰好是我们登录时登录接口返回的json数据的token部分，所以这个参数很有可能是判断我们是否登录的凭证，我们可以用postman模拟请求一下：

aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNjc1NTM5MC1hMWQ4MWY2NjA1NzA4ZjE0LnBuZw

通过postman的模拟请求发现如我们所料，我们只要在请求头里加上这个参数我们就可以获得对应的数据了。
解决了如何获得数据的问题，再来分析一下如何翻页，通过对比第一页和第二页提交的json数据可以发现几个关键参数，page、pagetotal、per_page分别代表当前请求页、记录总数、每页显示条数，因此根据pagetotal和per_page我们可以算出总的页数，到此整个项目分析结束，可以开始写程序了。

 

scrapy代码的编写

1.创建scrapy项目和爬虫：

  1. E:\>scrapy startproject ITjuzi
  2. E:\>scrapy genspider juzi itjuzi.com

 

2.编写items.py:

  1. # -*- coding: utf-8 -*-
  3. # Define here the models for your scraped items
  4. #
  5. # See documentation in:
  6. # https://doc.scrapy.org/en/latest/topics/items.html
  8. import scrapy
  11. class ItjuziItem(scrapy.Item):
  12. # define the fields for your item here like:
  13. # name = scrapy.Field()
  14. invse_des = scrapy.Field()
  15. invse_title = scrapy.Field()
  16. money = scrapy.Field()
  17. name = scrapy.Field()
  18. prov = scrapy.Field()
  19. round = scrapy.Field()
  20. invse_time = scrapy.Field()
  21. city = scrapy.Field()
  22. com_registered_name = scrapy.Field()
  23. com_scope = scrapy.Field()
  24. invse_company = scrapy.Field()

 

3.编写Spider:

  1. import scrapy
  2. from itjuzi.settings import JUZI_PWD, JUZI_USER
  3. import json
  6. class JuziSpider(scrapy.Spider):
  7. name = ‘juzi’
  8. allowed_domains = [‘itjuzi.com’]
  10. def start_requests(self):
  11. “””
  12. 先登录桔子网
  13. “””
  14. url = “https://www.itjuzi.com/api/authorizations”
  15. payload = {“account”: JUZI_USER, “password”: JUZI_PWD}
  16. # 提交json数据不能用scrapy.FormRequest，需要使用scrapy.Request，然后需要method、headers参数
  17. yield scrapy.Request(url=url,
  18. method=“POST”,
  19. body=json.dumps(payload),
  20. headers={‘Content-Type’: ‘application/json’},
  21. callback=self.parse
  22. )
  24. def parse(self, response):
  25. # 获取Authorization参数的值
  26. token = json.loads(response.text)
  27. url = “https://www.itjuzi.com/api/investevents”
  28. payload = {
  29. “pagetotal”: 0, “total”: 0, “per_page”: 20, “page”: 1, “type”: 1, “scope”: “”, “sub_scope”: “”,
  30. “round”: [], “valuation”: [], “valuations”: “”, “ipo_platform”: “”, “equity_ratio”: [“”],
  31. “status”: “”, “prov”: “”, “city”: [], “time”: [], “selected”: “”, “location”: “”, “currency”: [],
  32. “keyword”: “”
  33. }
  34. yield scrapy.Request(url=url,
  35. method=“POST”,
  36. body=json.dumps(payload),
  37. meta={‘token’: token},
  38. # 把Authorization参数放到headers中
  39. headers={‘Content-Type’: ‘application/json’, ‘Authorization’: token[‘data’][‘token’]},
  40. callback=self.parse_info
  41. )
  42. def parse_info(self, response):
  43. # 获取传递的Authorization参数的值
  44. token = response.meta[“token”]
  45. # 获取总记录数
  46. total = json.loads(response.text)[“data”][“page”][“total”]
  47. # 因为每页20条数据，所以可以算出一共有多少页
  48. if type(int(total)/20) is not int:
  49. page = int(int(total)/20)+1
  50. else:
  51. page = int(total)/20
  53. url = “https://www.itjuzi.com/api/investevents”
  54. for i in range(1,page+1):
  55. payload = {
  56. “pagetotal”: total, “total”: 0, “per_page”: 20, “page”:i , “type”: 1, “scope”: “”, “sub_scope”: “”,
  57. “round”: [], “valuation”: [], “valuations”: “”, “ipo_platform”: “”, “equity_ratio”: [“”],
  58. “status”: “”, “prov”: “”, “city”: [], “time”: [], “selected”: “”, “location”: “”, “currency”: [],
  59. “keyword”: “”
  60. }
  61. yield scrapy.Request(url=url,
  62. method=“POST”,
  63. body=json.dumps(payload),
  64. headers={‘Content-Type’: ‘application/json’, ‘Authorization’: token[‘data’][‘token’]},
  65. callback=self.parse_detail
  66. )
  69. def parse_detail(self, response):
  70. infos = json.loads(response.text)[“data”][“data”]
  71. for i in infos:
  72. item = ItjuziItem()
  73. item[“invse_des”] = i[“invse_des”]
  74. item[“com_des”] = i[“com_des”]
  75. item[“invse_title”] = i[“invse_title”]
  76. item[“money”] = i[“money”]
  77. item[“com_name”] = i[“name”]
  78. item[“prov”] = i[“prov”]
  79. item[“round”] = i[“round”]
  80. item[“invse_time”] = str(i[“year”])+“-“+str(i[“year”])+“-“+str(i[“day”])
  81. item[“city”] = i[“city”]
  82. item[“com_registered_name”] = i[“com_registered_name”]
  83. item[“com_scope”] = i[“com_scope”]
  84. invse_company = []
  85. for j in i[“investor”]:
  86. invse_company.append(j[“name”])
  87. item[“invse_company”] = “,”.join(invse_company)
  88. yield item

 

4.编写PIPELINE：

  1. from itjuzi.settings import DATABASE_DB, DATABASE_HOST, DATABASE_PORT, DATABASE_PWD, DATABASE_USER
  2. import pymysql
  4. class ItjuziPipeline(object):
  5. def __init__(self):
  6. host = DATABASE_HOST
  7. port = DATABASE_PORT
  8. user = DATABASE_USER
  9. passwd = DATABASE_PWD
  10. db = DATABASE_DB
  11. try:
  12. self.conn = pymysql.Connect(host=host, port=port, user=user, passwd=passwd, db=db, charset=‘utf8’)
  13. except Exception as e:
  14. print(“连接数据库出错,错误原因%s”%e)
  15. self.cur = self.conn.cursor()
  17. def process_item(self, item, spider):
  18. params = [item[‘com_name’], item[‘com_registered_name’], item[‘com_des’], item[‘com_scope’],
  19. item[‘prov’], item[‘city’], item[’round’], item[‘money’], item[‘invse_company’],item[‘invse_des’],item[‘invse_time’],item[‘invse_title’]]
  20. try:
  21. com = self.cur.execute(
  22. ‘insert into juzi(com_name, com_registered_name, com_des, com_scope, prov, city, round, money, invse_company, invse_des, invse_time, invse_title)values (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)’,params)
  23. self.conn.commit()
  24. except Exception as e:
  25. print(“插入数据出错,错误原因%s” % e)
  26. return item
  28. def close_spider(self, spider):
  29. self.cur.close()
  30. self.conn.close()

 

5.编写settings.py

  1. # -*- coding: utf-8 -*-
  3. # Scrapy settings for itjuzi project
  4. #
  5. # For simplicity, this file contains only settings considered important or
  6. # commonly used. You can find more settings consulting the documentation:
  7. #
  8. # https://doc.scrapy.org/en/latest/topics/settings.html
  9. # https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
  10. # https://doc.scrapy.org/en/latest/topics/spider-middleware.html
  12. BOT_NAME = ‘itjuzi’
  14. SPIDER_MODULES = [‘itjuzi.spiders’]
  15. NEWSPIDER_MODULE = ‘itjuzi.spiders’
  18. # Crawl responsibly by identifying yourself (and your website) on the user-agent
  19. #USER_AGENT = ‘itjuzi (+http://www.yourdomain.com)’
  21. # Obey robots.txt rules
  22. ROBOTSTXT_OBEY = False
  24. # Configure maximum concurrent requests performed by Scrapy (default: 16)
  25. #CONCURRENT_REQUESTS = 32
  27. # Configure a delay for requests for the same website (default: 0)
  28. # See https://doc.scrapy.org/en/latest/topics/settings.html#download-delay
  29. # See also autothrottle settings and docs
  30. DOWNLOAD_DELAY = 0.25
  31. # The download delay setting will honor only one of:
  32. #CONCURRENT_REQUESTS_PER_DOMAIN = 16
  33. #CONCURRENT_REQUESTS_PER_IP = 16
  35. # Disable cookies (enabled by default)
  36. #COOKIES_ENABLED = False
  38. # Disable Telnet Console (enabled by default)
  39. #TELNETCONSOLE_ENABLED = False
  41. # Override the default request headers:
  42. #DEFAULT_REQUEST_HEADERS = {
  43. # ‘Accept’: ‘text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8’,
  44. # ‘Accept-Language’: ‘en’,
  45. #}
  47. # Enable or disable spider middlewares
  48. # See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
  49. #SPIDER_MIDDLEWARES = {
  50. # ‘itjuzi.middlewares.ItjuziSpiderMiddleware’: 543,
  51. #}
  53. # Enable or disable downloader middlewares
  54. # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
  55. DOWNLOADER_MIDDLEWARES = {
  56. # ‘itjuzi.middlewares.ItjuziDownloaderMiddleware’: 543,
  57. ‘itjuzi.middlewares.RandomUserAgent’: 102,
  58. ‘itjuzi.middlewares.RandomProxy’: 103,
  59. }
  61. # Enable or disable extensions
  62. # See https://doc.scrapy.org/en/latest/topics/extensions.html
  63. #EXTENSIONS = {
  64. # ‘scrapy.extensions.telnet.TelnetConsole’: None,
  65. #}
  67. # Configure item pipelines
  68. # See https://doc.scrapy.org/en/latest/topics/item-pipeline.html
  69. ITEM_PIPELINES = {
  70. ‘itjuzi.pipelines.ItjuziPipeline’: 100,
  71. }
  73. # Enable and configure the AutoThrottle extension (disabled by default)
  74. # See https://doc.scrapy.org/en/latest/topics/autothrottle.html
  75. #AUTOTHROTTLE_ENABLED = True
  76. # The initial download delay
  77. #AUTOTHROTTLE_START_DELAY = 5
  78. # The maximum download delay to be set in case of high latencies
  79. #AUTOTHROTTLE_MAX_DELAY = 60
  80. # The average number of requests Scrapy should be sending in parallel to
  81. # each remote server
  82. #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
  83. # Enable showing throttling stats for every response received:
  84. #AUTOTHROTTLE_DEBUG = False
  86. # Enable and configure HTTP caching (disabled by default)
  87. # See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
  88. #HTTPCACHE_ENABLED = True
  89. #HTTPCACHE_EXPIRATION_SECS = 0
  90. #HTTPCACHE_DIR = ‘httpcache’
  91. #HTTPCACHE_IGNORE_HTTP_CODES = []
  92. #HTTPCACHE_STORAGE = ‘scrapy.extensions.httpcache.FilesystemCacheStorage’
  93. JUZI_USER = “1871111111111”
  94. JUZI_PWD = “123456789”
  96. DATABASE_HOST = ‘数据库ip’
  97. DATABASE_PORT = 3306
  98. DATABASE_USER = ‘数据库用户名’
  99. DATABASE_PWD = ‘数据库密码’
  100. DATABASE_DB = ‘数据表’
  102. USER_AGENTS = [
  103. “Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)”,
  104. “Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)”,
  105. “Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)”,
  106. “Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)”,
  107. “Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6”,
  108. “Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1”,
  109. “Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0”,
  110. “Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5”,
  111. “Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.29 Safari/537.36”
  112. ]
  114. PROXIES = [
  115. {‘ip_port’: ‘代理ip:代理IP端口’, ‘user_passwd’: ‘代理ip用户名:代理ip密码’},
  116. {‘ip_port’: ‘代理ip:代理IP端口’, ‘user_passwd’: ‘代理ip用户名:代理ip密码’},
  117. {‘ip_port’: ‘代理ip:代理IP端口’, ‘user_passwd’: ‘代理ip用户名:代理ip密码’},
  118. ]

 

6.让项目跑起来：

E:\>scrapy crawl juzi

7.结果展示:

aHR0cHM6Ly91cGxvYWQtaW1hZ2VzLmppYW5zaHUuaW8vdXBsb2FkX2ltYWdlcy8xNjc1NTM5MC03ODBhOTVhYWE3OTYyMzAzLnBuZw

PS：详情信息这里没有爬取，详情信息主要是根据上面列表页返回的json数据中每个公司的id来爬取，详情页的数据可以不用登录就能拿到如：https://www.itjuzi.com/api/investevents/10262327，https://www.itjuzi.com/api/get_investevent_down/10262327，还有最重要的一点是如果你的账号不是vip会员的话只能爬取前3页数据这个有点坑，其他的信息模块也是一样的分析方法，需要的可以自己去分析爬取。

https://blog.csdn.net/sandorn/article/details/104284233