如何在python中不用分割文件的使用多进程完成文件处理,换句话说,和单进程跑程序一样的完成处理,具体demo 如下
- def read_line(line):
- """
- 处理单行数据,业务逻辑代码,该demo在每行开始添加hello字段
- """
- line = "{}\t{}".format("hello",line)
- return line
-
-
- #读取输入信息
- lines = []
- with open("demo.txt",'r',encoding="utf-8") as fr:
- for line in fr:
- line = line.strip()
- lines.append(line)
-
-
- from multiprocessing import Pool
-
- with Pool(processes=10) as pool:
-
- ret = pool.map(read_line,lines)
-
- print(ret)
-
-
先把数据读入list ,再多进程处理,最后根据自己的要求完成结果的打印