因?yàn)殚_(kāi)發(fā)環(huán)境都是2.7的,所以不得不放棄python3,臨時(shí)用python2來(lái)開(kāi)發(fā)一些腳本.所以這里回顧了下python2的多進(jìn)程開(kāi)發(fā).這里整理了幾個(gè)小栗子. 希望可以解決問(wèn)題.
multiprocessing
multiprocessing模塊就是跨平臺(tái)版本的多進(jìn)程模塊。
- multiprocessing模塊提供了一個(gè)Process類(lèi)來(lái)代表一個(gè)進(jìn)程對(duì)象，
- 創(chuàng)建子進(jìn)程時(shí)抛蚤，只需要傳入一個(gè)執(zhí)行函數(shù)和函數(shù)的參數(shù)掸茅，創(chuàng)建一個(gè)Process實(shí)例又兵，用start()方法啟動(dòng).
- join()方法可以等待子進(jìn)程結(jié)束后再繼續(xù)往下運(yùn)行竖配，通常用于進(jìn)程間的同步哨鸭。
下面的例子演示了啟動(dòng)一個(gè)子進(jìn)程并等待其結(jié)束：
from multiprocessing import Process
import os
def TestFunc(name):
print "Get Child Process info:{}-{}".format(name,os.getpid())
if __name__ == '__main__':
p=Process(target=TestFunc,args=("TestFunc",))
p.start()
p.join()
print "---------END---------"
輸出:
Get Child Process info:TestFunc-10848
-------END-----------
Pool
如果要啟動(dòng)大量的子進(jìn)程通今，可以用進(jìn)程池的方式批量創(chuàng)建子進(jìn)程.
Pool的默認(rèn)大小是CPU的核數(shù)
import zipfile
import logging
import logging.handlers
import multiprocessing
import time
import os
log_filename="task_demo.log"
handler=logging.handlers.RotatingFileHandler(log_filename,maxBytes=20,backupCount=4)
def get_logger(name="root"):
logging.basicConfig(level=logging.DEBUG, format='%(relativeCreated)6d %(threadName)s %(message)s')
return logging.getLogger(name)
def TestFunc(filename):
print("start jobs")
print(filename)
time.sleep(1)
print("end jobs")
return True
if __name__ == '__main__':
logger=get_logger()
# logger.addHandler(handler)
logger.info("--------------START JOBS-------------------")
worker_num=20
pool=multiprocessing.Pool(processes=worker_num)
path=r"C:\Users\youxi\Downloads\Flask-User-starter-app-master\Flask-User-starter-app-master"
task=os.listdir(path)
for i in task:
pool.apply_async(TestFunc,args=(i,))
logger.info("waitting for all subprocess done...")
pool.close()
pool.join()
logger.info("All subprocess done!")
進(jìn)程間通訊
Process之間肯定是需要通信的粥谬，操作系統(tǒng)提供了很多機(jī)制來(lái)實(shí)現(xiàn)進(jìn)程間的通信。Python的multiprocessing模塊包裝了底層的機(jī)制辫塌，提供了Queue漏策、Pipes等多種方式來(lái)交換數(shù)據(jù)。
from multiprocessing import Process, Queue
import os, time, random
# 寫(xiě)數(shù)據(jù)進(jìn)程執(zhí)行的代碼:
def write(q):
for value in ['A', 'B', 'C']:
print 'Put %s to queue...' % value
q.put(value)
time.sleep(random.random())
# 讀數(shù)據(jù)進(jìn)程執(zhí)行的代碼:
def read(q):
while True:
value = q.get(True)
print 'Get %s from queue.' % value
if __name__=='__main__':
# 父進(jìn)程創(chuàng)建Queue璃氢，并傳給各個(gè)子進(jìn)程：
q = Queue()
pw = Process(target=write, args=(q,))
pr = Process(target=read, args=(q,))
# 啟動(dòng)子進(jìn)程pw哟玷，寫(xiě)入:
pw.start()
# 啟動(dòng)子進(jìn)程pr，讀取:
pr.start()
# 等待pw結(jié)束:
pw.join()
# pr進(jìn)程里是死循環(huán)一也，無(wú)法等待其結(jié)束巢寡，只能強(qiáng)行終止:
pr.terminate()
