【4】进程管理-13-并行化运行--pathos

pathos.multiprocesssing是multiprocesssing的替代,它fork的multiprocesssing,但是用dill来进行序列化处理,因此也能在类的方法中来进行多进程的处理

一、介绍

二、案例

例一:

#self.do_sample_qc(sample):
   pass

from pathos.multiprocessing import ProcessingPool as Pool
Pool(20).map(self.do_sample_qc, [sample for sample in self.chip])

例二:

from pathos import multiprocessing as mp
def analysis_pseudogene(args):
    pass

args_list = []
results = []
for k1 in primers_locs.keys():
   args_list.append(info)

pool = mp.ProcessingPool(mp.cpu_count()-5)
results.extend(pool.map(analysis_pseudogene, args_list))
pool.close()

例三:

多线程的处理,并保存数据

import multiprocessing as mp
from pathos import multiprocessing as pathos_mp
import gzip

example_data = range(100)
def process_point(point):
    output = "output-%d.gz" % mp.current_process().pid
    with gzip.open(output, "a+") as fout:
        fout.write('%d\n' % point**2)

pool = pathos_mp.Pool(8)
pool.map(process_point, example_data)

三、讨论

参考资料:

http://stackoverflow.com/questions/8804830/python-multiprocessing-pickling-error

个人公众号,比较懒,很少更新,可以在上面提问题:

更多精彩,请移步公众号阅读:

Sam avatar
About Sam
专注生物信息 专注转化医学