• 使用CompletionService进行多个文件打包为zip下载

    最近没怎么写博客了，因为前段时间在准备软考复习，昨天考完试，现在总算轻松一点了，有更多自由的时间了，总结一下JUC包下的一些并发工具类，主要是从使用场景入手。

    CompletionService可以用于实现任务并行化，提高任务处理的效率。以下是CompletionService在项目中的应用场景：

    1. 多个任务需要并行执行，但是它们的执行时间不同，需要优先处理先执行完成的任务。
    2. 需要获取多个任务的执行结果，但是不关心任务的执行顺序。
    3. 处理任务的线程池数量有限，需要充分利用线程池资源，提高任务处理的效率。

    举例来说，假设有一个需要下载多个文件的任务，使用CompletionService可以将文件下载任务分配给多个线程并行执行，而不需要等待所有文件都下载完成才开始处理文件。同时，由于每个线程的下载时间可能不同，CompletionService可以优先获取下载完成的文件，并在下载完成后立即处理文件，提高任务处理的效率。

    以下就是使用CompletionService从minio文件服务器下载文件，并进行打包压缩的使用场景；因为我们对文件进行打包压缩，并不关心下载的多个文件的下载顺序，哪个文件先下载完，就先处理哪个文件，然后最后统一放入到一个文件夹进行打包压缩，提供下载

    业务场景如下：

    需求是选中多个附件，然后批量下载，下载下来后是一个zip文件，附件使用的是minio文件存储服务

    业务实现代码如下：

    1. import cn.hutool.core.io.FileUtil;
    2. import com.baomidou.mybatisplus.core.toolkit.Wrappers;
    3. import com.baomidou.mybatisplus.extension.plugins.pagination.Page;
    4. import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;
    5. import com.xiaomifeng.common.mapper.AttachmentMidMapper;
    6. import com.xiaomifeng.common.service.AttachmentMidService;
    7. import com.xiaomifeng.common.util.CaseFormatUtil;
    8. import com.xiaomifeng.entity.AttachmentMid;
    9. import com.xiaomifeng.minio.configuration.MinioConfig;
    10. import com.xiaomifeng.minio.entity.MinioFile;
    11. import com.xiaomifeng.minio.service.MinioFileService;
    12. import com.xiaomifeng.util.MinioClientUtils;
    13. import com.xiaomifeng.util.ZipUtil;
    14. import com.google.common.collect.Lists;
    15. import com.google.common.util.concurrent.ThreadFactoryBuilder;
    16. import lombok.Setter;
    17. import org.apache.commons.collections4.CollectionUtils;
    18. import org.apache.commons.collections4.MapUtils;
    19. import org.apache.commons.lang3.RandomStringUtils;
    20. import org.apache.commons.lang3.StringUtils;
    21. import org.apache.commons.lang3.SystemUtils;
    22. import org.springframework.beans.factory.annotation.Autowired;
    23. import org.springframework.security.concurrent.DelegatingSecurityContextExecutorService;
    24. import org.springframework.stereotype.Service;
    25.  
    26. import javax.servlet.http.HttpServletResponse;
    27. import java.io.File;
    28. import java.io.IOException;
    29. import java.io.InputStream;
    30. import java.io.OutputStream;
    31. import java.util.ArrayList;
    32. import java.util.List;
    33. import java.util.Map;
    34. import java.util.concurrent.*;
    35.  
    36. /**
    37.  * @author xiaomifeng1010
    38.  * @version 1.0
    39.  * @date: 
    40.  * @Description
    41.  */
    42. @Service
    43. @Setter(onMethod_ = @Autowired)
    44. public class AttachmentMidServiceImpl extends ServiceImpl implements AttachmentMidService {
    45.  
    46.     private MinioClientUtils minioClientUtils;
    47.  
    48.     private MinioConfig minioConfig;
    49.  
    50.     private MinioFileService minioFileService;
    51.  
    52.  
    53.     @Override
    54.     public void export2Zip(List fileIds, HttpServletResponse response) throws IOException {
    55.         if (CollectionUtils.isNotEmpty(fileIds)) {
    56.             List fileList = new ArrayList<>();
    57.             ThreadFactory threadFactory = new ThreadFactoryBuilder().setNameFormat("zip-file-%d").build();
    58.             ExecutorService threadPoolExecutor = new ThreadPoolExecutor(12, 20, 0L, TimeUnit.SECONDS, new LinkedBlockingQueue<>(100),
    59.                     threadFactory, new ThreadPoolExecutor.CallerRunsPolicy());
    60.             threadPoolExecutor = new DelegatingSecurityContextExecutorService(threadPoolExecutor);
    61.             CompletionService completionService = new ExecutorCompletionService<>(threadPoolExecutor);
    62.             List> callableList = Lists.newArrayList();
    63.             fileIds.forEach(fileId -> {
    64.                 MinioFile minioFile = minioFileService.getById(fileId);
    65.                 if (minioFile != null) {
    66.                     callableList.add(() -> {
    67.                         String fileName = minioFile.getFileName();
    68.                         String originalFileName = minioFile.getOriginalFileName();
    69.                         originalFileName = StringUtils.substringBeforeLast(originalFileName, ".");
    70.                         String fileExtName = minioFile.getFileExtName();
    71.                         Integer access = minioFile.getAccess();
    72.                         String bucketName = minioConfig.getBucketName();
    73.  
    74.                         if (access == 2) {
    75.                             bucketName = minioConfig.getBucketNamePrivate();
    76.                         }
    77.                         try {
    78.                             InputStream inputStream = minioClientUtils.getObject(bucketName, fileName);
    79.                             if (inputStream != null) {
    80.                                 File javaIoTmpDir = SystemUtils.getJavaIoTmpDir();
    81.                                 String finalFilePath = javaIoTmpDir.getAbsolutePath() + File.separator + originalFileName + "." + fileExtName;
    82.                                 File newFile = FileUtil.newFile(finalFilePath);
    83.                                 if (newFile.exists()) {
    84.                                     originalFileName = originalFileName + RandomStringUtils.randomNumeric(5);
    85.                                 }
    86.                                 if (StringUtils.length(originalFileName) < 3) {
    87.                                     originalFileName = originalFileName + RandomStringUtils.randomNumeric(3);
    88.                                 }
    89.                                 File tempFile = FileUtil.createTempFile(originalFileName, "." + fileExtName, javaIoTmpDir, true);
    90.                                 tempFile = FileUtil.writeFromStream(inputStream, tempFile);
    91.                                 return tempFile;
    92.                             }
    93.                         } catch (Exception e) {
    94.                             log.error("从minio获取文件失败：", e);
    95.                         }
    96.                         return null;
    97.                     });
    98.  
    99.  
    100.                 }
    101.             });
    102.             if (CollectionUtils.isNotEmpty(callableList)) {
    103.                 for (Callable fileCallable : callableList) {
    104.                     completionService.submit(fileCallable);
    105.  
    106.                 }
    107.  
    108.                 for (int i = 0; i < callableList.size(); i++) {
    109.  
    110.                     try {
    111.                         File temFile = completionService.take().get();
    112.                         if (temFile != null) {
    113.                             fileList.add(temFile);
    114.                         }
    115.  
    116.                     } catch (InterruptedException e) {
    117.                         Thread.currentThread().interrupt();
    118.                     } catch (ExecutionException e) {
    119.                         log.error("completionService获取文件对象失败:{}", e);
    120.                     }
    121.  
    122.                 }
    123.  
    124.             }
    125.             threadPoolExecutor.shutdown();
    126.             String zipFileName = "附件管理";
    127.             File tempZipFile = FileUtil.createTempFile(zipFileName, ".zip", true);
    128.  
    129.             response.setContentType("application/zip");
    130.             response.setHeader("Location", tempZipFile.getName());
    131.             response.setHeader("Content-Disposition", "attachment; filename=" + tempZipFile.getName());
    132.             OutputStream outputStream = response.getOutputStream();
    133.             ZipUtil.filesToZip(fileList, outputStream);
    134.             tempZipFile.delete();
    135.             outputStream.flush();
    136.             outputStream.close();
    137. //            清理临时文件
    138.             fileList.forEach(File::delete);
    139.  
    140.         }
    141.  
    142.     }
    143.  
    144.  
    145.  
    146. }

    minio工具类，可以参照我的另一篇安装使用minio的博文中，有非常详细的用法MinIO文件服务器，从安装到使用

    还有一个ziputil工具类

    1. import java.io.File;
    2. import java.io.FileInputStream;
    3. import java.io.FileNotFoundException;
    4. import java.io.FileOutputStream;
    5. import java.io.IOException;
    6. import java.io.InputStream;
    7. import java.io.OutputStream;
    8. import java.nio.charset.Charset;
    9. import java.util.ArrayList;
    10. import java.util.Enumeration;
    11. import java.util.Iterator;
    12. import java.util.List;
    13. import java.util.zip.ZipEntry;
    14. import java.util.zip.ZipFile;
    15. import java.util.zip.ZipOutputStream;
    16. import org.slf4j.Logger;
    17. import org.slf4j.LoggerFactory;
    18.  
    19.  
    20. @Slf4j
    21. public class ZipUtil {
    22.    
    23.     private static String destFileName;
    24.  
    25.     public static List unZipFiles(File zipFile, String descDir) throws IOException {
    26.         List fileNames = new ArrayList();
    27.         ZipFile zip = new ZipFile(zipFile, Charset.forName("GBK"));
    28.         String name = zip.getName().substring(zip.getName().lastIndexOf(92) + 1, zip.getName().lastIndexOf(46));
    29.         File pathFile = new File(descDir + name);
    30.         if (!pathFile.exists()) {
    31.             pathFile.mkdirs();
    32.         }
    33.  
    34.         String outPath = "";
    35.         Enumeration entries = zip.entries();
    36.  
    37.         while(true) {
    38.             InputStream in;
    39.             do {
    40.                 if (!entries.hasMoreElements()) {
    41.                     pathFile.delete();
    42.                     return fileNames;
    43.                 }
    44.  
    45.                 ZipEntry entry = (ZipEntry)entries.nextElement();
    46.                 String zipEntryName = entry.getName();
    47.                 fileNames.add(zipEntryName);
    48.                 in = zip.getInputStream(entry);
    49.                 outPath = (descDir + name + "/" + zipEntryName).replaceAll("\\*", "/");
    50.                 File file = new File(outPath.substring(0, outPath.lastIndexOf(47)));
    51.                 if (!file.exists()) {
    52.                     file.mkdirs();
    53.                 }
    54.             } while((new File(outPath)).isDirectory());
    55.  
    56.             FileOutputStream out = new FileOutputStream(outPath);
    57.             byte[] buf1= new byte[1024];
    58.  
    59.             int len;
    60.             while((len = in.read(buf1)) > 0) {
    61.                 out.write(buf1, 0, len);
    62.             }
    63.  
    64.             in.close();
    65.             out.close();
    66.         }
    67.     }
    68.  
    69.     public static void docToZip(String srcDir, String targetFile, boolean KeepDirStructure) throws RuntimeException, FileNotFoundException {
    70.         long start = System.currentTimeMillis();
    71.         ZipOutputStream zos = null;
    72.         FileOutputStream out = new FileOutputStream(new File(targetFile));
    73.  
    74.         try {
    75.             zos = new ZipOutputStream(out);
    76.             File sourceFile = new File(srcDir);
    77.             destFileName = targetFile.substring(targetFile.lastIndexOf(File.separator) + 1, targetFile.length());
    78.             System.out.println(destFileName);
    79.             compress(sourceFile, zos, sourceFile.getName(), KeepDirStructure);
    80.             long end = System.currentTimeMillis();
    81.             System.out.println("压缩完成，耗时：" + (end - start) + " ms");
    82.         } catch (Exception var20) {
    83.             throw new RuntimeException("zip error from ZipUtils", var20);
    84.         } finally {
    85.             if (zos != null) {
    86.                 try {
    87.                     zos.close();
    88.                 } catch (IOException var19) {
    89.                     log.error(var19.getMessage(), var19);
    90.                 }
    91.             }
    92.  
    93.             if (out != null) {
    94.                 try {
    95.                     out.close();
    96.                 } catch (IOException var18) {
    97.                     log.error(var18.getMessage(), var18);
    98.                 }
    99.             }
    100.  
    101.         }
    102.  
    103.     }
    104.  
    105.     public static void filesToZip(List srcFiles, OutputStream out) throws RuntimeException {
    106.         long start = System.currentTimeMillis();
    107.         ZipOutputStream zos = null;
    108.         short BUFFER_SIZE = 2048;
    109.  
    110.         try {
    111.             zos = new ZipOutputStream(out);
    112.             Iterator var6 = srcFiles.iterator();
    113.  
    114.             while(var6.hasNext()) {
    115.                 File srcFile = (File)var6.next();
    116.                 byte[] buf = new byte[BUFFER_SIZE];
    117.                 zos.putNextEntry(new ZipEntry(srcFile.getName()));
    118.                 FileInputStream in = new FileInputStream(srcFile);
    119.  
    120.                 int len;
    121.                 while((len = in.read(buf)) != -1) {
    122.                     zos.write(buf, 0, len);
    123.                 }
    124.  
    125.                 zos.closeEntry();
    126.                 in.close();
    127.             }
    128.  
    129.             long end = System.currentTimeMillis();
    130.             System.out.println("压缩完成，耗时：" + (end - start) + " ms");
    131.         } catch (Exception var18) {
    132.             throw new RuntimeException("zip error from ZipUtils", var18);
    133.         } finally {
    134.             if (zos != null) {
    135.                 try {
    136.                     zos.close();
    137.                 } catch (IOException var17) {
    138.                     log.error(var17.getMessage(), var17);
    139.                 }
    140.             }
    141.  
    142.         }
    143.     }
    144.  
    145.     private static void compress(File sourceFile, ZipOutputStream zos, String name, boolean KeepDirStructure) throws Exception {
    146.         int BUFFER_SIZE = 2048;
    147.         byte[] buf = new byte[BUFFER_SIZE];
    148.         System.out.println(name);
    149.         if (sourceFile.isFile()) {
    150.             if (!destFileName.equals(sourceFile.getName())) {
    151.                 zos.putNextEntry(new ZipEntry(name));
    152.                 FileInputStream in = new FileInputStream(sourceFile);
    153.  
    154.                 int len;
    155.                 while((len = in.read(buf)) != -1) {
    156.                     zos.write(buf, 0, len);
    157.                 }
    158.  
    159.                 zos.closeEntry();
    160.                 in.close();
    161.             }
    162.         } else {
    163.             File[] listFiles = sourceFile.listFiles();
    164.             if (listFiles != null && listFiles.length != 0) {
    165.                 File[] var12 = listFiles;
    166.                 int var8 = listFiles.length;
    167.  
    168.                 for(int var9 = 0; var9 < var8; ++var9) {
    169.                     File file = var12[var9];
    170.                     if (KeepDirStructure) {
    171.                         compress(file, zos, name + "/" + file.getName(), KeepDirStructure);
    172.                     } else {
    173.                         compress(file, zos, file.getName(), KeepDirStructure);
    174.                     }
    175.                 }
    176.             } else if (KeepDirStructure) {
    177.                 zos.putNextEntry(new ZipEntry(name + "/"));
    178.                 zos.closeEntry();
    179.             }
    180.         }
    181.  
    182.     }
    183. }
    184.  
    185.

    CompletionService调用线程池异步从minio下载文件，下载好的文件放到List集合，然后使用ziputil进行 压缩，有个注意事项，就是在创建临时文件的时候，文件名的字符长度不能小于3，否则会抛出异常。所以在代码中有个文件名字符长度的判断。

  • 相关阅读:
    好消息，完整版的jwt鉴权机制来了呦
    【全志T113-S3_100ask】5-编写按键驱动（input子系统+内核按键驱动）
    增量与存量，谈投资与市场的规模与人类的未来
    什么是哈希表？如何使用哈希表进行数据存储和查找？
    计算机三级数据库后台编程技术练习题（一）
    web网页设计期末课程大作业：旅游网页主题网站设计——中国风的温泉酒店预订网(13页)HTML+CSS+JavaScript
    vue 3 第三十五章：集成 tailwind Css
    python：flatten()参数详解
    网络安全人才发展史
    crypto加密
  • 原文地址：https://blog.csdn.net/u011174699/article/details/130911870