这里主要想记录下今天碰到的一个小知识点：Python中的并行编程速率如何？

　　我想把AutoTool做一个并行化改造，主要目的当然是想提高多任务的执行速度。第一反应就是想到用多线程执行不同模块任务，但是在我收集Python多线程编程资料的时候发现一个非常奇怪的信息，那就是Python的多线程并不是真正的多线程，因为有一个GIL的存在（可以参考这篇文章讲解《Python最难的问题》）导致Python实际上默认（CPython解释器）只能是单线程执行。

　　这里我写了一个例子可以看看：

 #!/usr/bin/env python

 # -*- coding: utf-8 -*-

 # @File    : batch_swig_runner.py

 # @Time    : 2019/7/8 18:09

 # @Author  : KuLiuheng

 # @Email   : liuheng.klh@alibaba-inc.com

 from swig_runner import SwigRunner

 import time

 import logging

 from threading import Thread

 from multiprocessing import Pool

 class TestRunner(Thread):

     def __init__(self, name, path):

         super(TestRunner, self).__init__()

         self.name = name

         self.path = path

     def run(self):

         logging.warning("Message from the thread-%s START" % self.name)

         for i in range(10000000):   # 耗时操作模拟

             j = int(i) * 10.1

         # time.sleep(1)

         logging.warning("Message from the thread-%s END" % self.name)

         return self.path

 def multi_process(mname, mpath):

     logging.warning("Message from the thread-%s START" % mname)

     for i in range(10000000):   # 耗时操作模拟

         j = int(i) * 10.1

     # time.sleep(1)

     logging.warning("Message from the thread-%s END" % mname)

 class BatchSwigRunner(object):

     def __init__(self, modules=None):

         """

         用模块信息字典（工程名: 工程路径）来初始化

         :param modules: {工程名: 工程路径}

         """

         if modules is not None:

             self._modules = modules

         else:

             self._modules = dict()

     def add_module_info(self, name, path):

         self._modules[name] = path

     def start(self):

         """

         启动批量任务执行，并返回执行过程中的错误信息

         :return: list(工程序号，工程名称) 出错的工程信息列表

         """

         runners = list()

         for (project_name, project_path) in self._modules.items():

             # logging.warning('BatchSwigRunner.start() [%s][%s]' % (project_name, project_path))

             sub_runner = TestRunner(project_name, project_path)

             sub_runner.daemon = True

             sub_runner.start()

             runners.append(sub_runner)

         for runner in runners:

             runner.join()

 if __name__ == '__main__':

     batch_runner = BatchSwigRunner()

     batch_runner.add_module_info('name1', 'path1')

     batch_runner.add_module_info('name2', 'path2')

     batch_runner.add_module_info('name3', 'path3')

     batch_runner.add_module_info('name4', 'path4')

     start_time = time.time()

     batch_runner.start()

     print 'Total time comsumed = %.2fs' % (time.time() - start_time)

     print('========================================')

     start_time = time.time()

     for index in range(4):

         logging.warning("Message from the times-%d START" % index)

         for i in range(10000000):       # 耗时操作模拟

             j = int(i) * 10.1

         # time.sleep(1)

         logging.warning("Message from the times-%d END" % index)

     print '>>Total time comsumed = %.2fs' % (time.time() - start_time)

     print('----------------------------------------------')

     start_time = time.time()

     pool = Pool(processes=4)

     for i in range(4):

         pool.apply_async(multi_process, ('name++%d' % i, 'path++%d' % i))

     pool.close()

     pool.join()

     print '>>>> Total time comsumed = %.2fs' % (time.time() - start_time)

　　看结果就发现很神奇的结论：

C:\Python27\python.exe E:/VirtualShare/gitLab/GBL-310/GBL/AutoJNI/autoTool/common/batch_swig_runner.py

WARNING:root:Message from the thread-name4 START

WARNING:root:Message from the thread-name2 START

WARNING:root:Message from the thread-name3 START

WARNING:root:Message from the thread-name1 START

WARNING:root:Message from the thread-name2 END

WARNING:root:Message from the thread-name4 END

WARNING:root:Message from the thread-name3 END

Total time comsumed = 15.92s

========================================

WARNING:root:Message from the thread-name1 END

WARNING:root:Message from the times-0 START

WARNING:root:Message from the times-0 END

WARNING:root:Message from the times-1 START

WARNING:root:Message from the times-1 END

WARNING:root:Message from the times-2 START

WARNING:root:Message from the times-2 END

WARNING:root:Message from the times-3 START

WARNING:root:Message from the times-3 END

>>Total time comsumed = 11.59s

----------------------------------------------

WARNING:root:Message from the thread-name++0 START

WARNING:root:Message from the thread-name++1 START

WARNING:root:Message from the thread-name++2 START

WARNING:root:Message from the thread-name++3 START

WARNING:root:Message from the thread-name++1 END

WARNING:root:Message from the thread-name++0 END

WARNING:root:Message from the thread-name++2 END

WARNING:root:Message from the thread-name++3 END

>>>> Total time comsumed = 5.69s

Process finished with exit code 0

　　其运行速度是（计算密集型）：multiprocessing > normal > threading.Thread

　　请注意这里用的是持续计算来模拟耗时操作：

for i in range(10000000):   # 耗时操作模拟

    j = int(i) * 10.1

　　如果用空等待（time.sleep(1)类似IO等待）来模拟耗时操作，那么结果就是（IO等待型）：threading.Thread > multiprocessing > normal

C:\Python27\python.exe E:/VirtualShare/gitLab/GBL-310/GBL/AutoJNI/autoTool/common/batch_swig_runner.py

WARNING:root:Message from the thread-name4 START

WARNING:root:Message from the thread-name2 START

WARNING:root:Message from the thread-name3 START

WARNING:root:Message from the thread-name1 START

WARNING:root:Message from the thread-name3 END

WARNING:root:Message from the thread-name4 END

WARNING:root:Message from the thread-name2 END

WARNING:root:Message from the thread-name1 END

WARNING:root:Message from the times-0 START

Total time comsumed = 1.01s

========================================

WARNING:root:Message from the times-0 END

WARNING:root:Message from the times-1 START

WARNING:root:Message from the times-1 END

WARNING:root:Message from the times-2 START

WARNING:root:Message from the times-2 END

WARNING:root:Message from the times-3 START

WARNING:root:Message from the times-3 END

>>Total time comsumed = 4.00s

----------------------------------------------

WARNING:root:Message from the thread-name++0 START

WARNING:root:Message from the thread-name++1 START

WARNING:root:Message from the thread-name++2 START

WARNING:root:Message from the thread-name++3 START

WARNING:root:Message from the thread-name++0 END

WARNING:root:Message from the thread-name++1 END

WARNING:root:Message from the thread-name++2 END

WARNING:root:Message from the thread-name++3 END

>>>> Total time comsumed = 1.73s

Process finished with exit code 0

　　为何会有这样的结果呢？

（1）threading机制中因为GIL的存在，实际上是一把全局锁让多线程变成了CPU线性执行，只可能用到一颗CPU计算。当sleep这样是释放CPU操作发生时，可以迅速切换线程，切换速度可以接受（比multiprocessing快），比normal（阻塞等待）当然快的多；

（2）这里用了多进程Pool，可以真正意义上使用多CPU，对于CPU计算密集型的操作（上面的for循环计算）那么肯定是多核比单核快。所以就出现了第一种测试场景的结果。

最新文章

  1. 学习FFmpeg API
  2. iOS----关于第三方的运用（有待补充）
  3. 他们在军训，我在搞 OI（四）
  4. transient的理解
  5. Python之定向爬虫Scrapy
  6. Web前端新人笔记之jquery入门
  7. 01-Objective-C
  8. C#中的四舍五入算法
  9. 【Xamarin开发IOS-IOS生命周期】
  10. 面试题-Java基础-垃圾回收
  11. app后端设计(7)-- 项目管理
  12. 一篇文章搞懂Android组件化
  13. python 实现进制转换（二进制转十进制）
  14. django 富文本编辑器
  15. Django Context对象 + 过滤器 + 标签
  16. SSIS 你真的了解事务吗？
  17. ERP渠道管理添加验证和查询（二十二）
  18. urlparse 用法
  19. 2016-2017-20155329 《Java程序设计》第十周学习总结
  20. 用原生JS实现一个轮播（包含全部代码和详细思路）

热门文章

  1. CF837D Round Subset 动态规划
  2. mysql5.7版本以上下载安装
  3. 1829：【02NOIP提高组】自由落体
  4. linux下redis的安装、启动、关闭和卸载
  5. zoom:1的常见作用
  6. 【hadoop环境问题】namenode无法启动问题解决
  7. Java-JVM 锁优化
  8. RK3399 修改系统默认语言为简体中文
  9. 连接池设置导致的“血案” 原创： 一页破书 一页破书 5月6日 这个问题被投诉的几个月了，一直没重视——内部客户嘛😿 问题现象： 隔几周就会出现 A服务调用B服务超时 脚趾头想就是防火墙的问题，A、B两服务之间有防火墙 找运维查看防火墙日志确实断掉了tcp连接，但是是因为B服务5分钟没有回包，下面这个表情就是我当时的心情——其实我们在防火墙、A服务、B服务都抓包了，几十个G的t
  10. 【转】nodejs获取post请求发送的formData数据