一、UA池和代理池

1、UA池

scrapy的下载中间件：

下载中间件（Downloader Middlewares）位于scrapy引擎和下载器之间的一层组件。

作用：

（1）引擎将请求传递给下载器过程中，下载中间件可以对请求进行一系列处理。比如设置请求的 User-Agent，设置代理等

（2）在下载器完成将Response传递给引擎中，下载中间件可以对响应进行一系列处理。比如进行gzip解压等。

我们主要使用下载中间件处理请求，一般会对请求设置随机的User-Agent ，设置随机的代理。目的在于防止爬取网站的反爬虫策略。

UA池作用：尽可能多的将scrapy框架中的请求伪装成不同类型的浏览器以及不同电脑的身份。

- 操作流程：

1.下载中间件中拦截请求

2.将拦截到的请求的请求头信息中的UA进行伪装

3.在配置文件中开启下载中间件

代码展示：（只需要看proccess_request方法）

# -*- coding: utf-8 -*-

# Define here the models for your spider middleware

#

# See documentation in:

# https://docs.scrapy.org/en/latest/topics/spider-middleware.html

from scrapy import signals

from scrapy.http import HtmlResponse

import random

from time import sleep

class WangyixinwenDownloaderMiddleware(object):

    # Not all methods need to be defined. If a method is not defined,

    # scrapy acts as if the downloader middleware does not modify the

    # passed objects.

    user_agent_list = [

        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 "

        "(KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1",

        "Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 "

        "(KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11",

        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 "

        "(KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6",

        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 "

        "(KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6",

        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.1 "

        "(KHTML, like Gecko) Chrome/19.77.34.5 Safari/537.1",

        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 "

        "(KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5",

        "Mozilla/5.0 (Windows NT 6.0) AppleWebKit/536.5 "

        "(KHTML, like Gecko) Chrome/19.0.1084.36 Safari/536.5",

        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "

        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

        "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/536.3 "

        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/536.3 "

        "(KHTML, like Gecko) Chrome/19.0.1063.0 Safari/536.3",

        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "

        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",

        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "

        "(KHTML, like Gecko) Chrome/19.0.1062.0 Safari/536.3",

        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "

        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

        "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 "

        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

        "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 "

        "(KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3",

        "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 "

        "(KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3",

        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 "

        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24",

        "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 "

        "(KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24"

    ]

    # 代理池

    PROXY_http = [

        '153.180.102.104:80',

        '195.208.131.189:56055',

    ]

    PROXY_https = [

        '120.83.49.90:9000',

        '95.189.112.214:35508',

    ]

    def process_request(self, request, spider):

        """

        在这里可以设置UA，如果要设置IP就需要等自己IP出现问题再前去process_exception中设置代理IP

        但最后一定要返回request对象

        :param request:

        :param spider:

        :return:

        """

        # ua = random.choice(self.user_agent_list)#随机选择一个元素（反爬策略（即不同浏览器和机型））出来

        # request.headers["User-Agent"] = ua

        #return request#将修正的request对象重新进行发送

        return None

    def process_response(self, request, response, spider):

        #刚才响应体缺失，因此从这里我们应该重新返回新的响应体

        #这里要用到爬虫程序中的urls，判断url是否在里面，在urls里面的就会出现响应缺失，

        # 、因此需要返回新的响应体

        if request.url in spider.urls:

            #响应缺失是因为是动态加载数据，因此我们配合selenium使用

            #在这里实例化selenium的话会被实例化多次，然而selenium只需要实例化一次，

            #这个时候我们可以将selenium放在实例化一次的爬虫程序开始的时候,实例化完成引入

            sleep(2)

            bro = spider.bro.get(url=request.url)#浏览器中发送请求

            sleep(1)

            spider.bro.execute_script("window.scrollTo(0,document.body.scrollHeight)")

            sleep(1.5)

            spider.bro.execute_script("window.scrollTo(0,document.body.scrollHeight)")

            sleep(0.7)

            spider.bro.execute_script("window.scrollTo(0,document.body.scrollHeight)")

            sleep(1)

            spider.bro.execute_script("window.scrollTo(0,document.body.scrollHeight)")

            #发送到请求我们需要获取浏览器当前页面的源码数据 获取数据之前需要翻滚页面

            page_text = spider.bro.page_source

            #改动返回响应对象  scrapy提供了一个库url=spider.bro.current_url, body=page_text, encoding='utf-8', request=request

            new_response = HtmlResponse(url=request.url,body=page_text,encoding="utf-8",request=request)

            return new_response

        else:

            return response

        #提交完新的响应体之后，去设置将下载中间件打开

    def process_exception(self, request, exception, spider):

        # Called when a download handler or a process_request()

        # (from other downloader middleware) raises an exception.

        # Must either:

        # - return None: continue processing this exception

        # - return a Response object: stops process_exception() chain

        # - return a Request object: stops process_exception() chain

        #如果抛出异常，那么就换代理IP

        # 代理ip的设定

        # if request.url.split(':')[0] == 'http':

        #     request.meta['proxy'] = random.choice(self.PROXY_http)

        # else:

        #     request.meta['proxy'] = random.choice(self.PROXY_https)

        # # 将修正后的请求对象重新进行请求发送

        # return request

        pass

然后去配置文件开启中间件settings.py:

# Enable or disable downloader middlewares

# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html

DOWNLOADER_MIDDLEWARES = {

   'Wangyixinwen.middlewares.WangyixinwenDownloaderMiddleware': 543,

}

2、代理池

作用：尽可能多的将scrapy框架中的请求的IP设置成不同的，当服务器禁止自己电脑的ip的时候将自动调用process_exception中的ip代理设置

操作流程：

1.在下载中间件中拦截请求

2.将拦截到的请求的IP修改成某一代理IP

3.在配置文件中开启下载中间件

 def process_exception(self, request, exception, spider):

        # Called when a download handler or a process_request()

        # (from other downloader middleware) raises an exception.

        # Must either:

        # - return None: continue processing this exception

        # - return a Response object: stops process_exception() chain

        # - return a Request object: stops process_exception() chain

        #如果抛出异常，那么就换代理IP

        # 代理ip的设定

         if request.url.split(':')[0] == 'http':

             request.meta['proxy'] = random.choice(self.PROXY_http)

         else:

             request.meta['proxy'] = random.choice(self.PROXY_https)

         # 将修正后的请求对象重新进行请求发送

         return request

# Enable or disable downloader middlewares

# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html

DOWNLOADER_MIDDLEWARES = {

   'Wangyixinwen.middlewares.WangyixinwenDownloaderMiddleware': 543,

}

巴特西

12-UA池和代理池

一、UA池和代理池

1、UA池

2、代理池

拦截请求直接在上面中间件代码的process_exception中，配置文件也在上面。

最新文章

热门文章