造轮子之文件管理

合集 - asp.net core入门到造轮子(34)

1.asp.net core入门到造轮子-目录07-242.asp.net core之Startup07-243.asp.net core之依赖注入07-254.asp.net core之中间件07-265.asp.net core之Host07-276.asp.net core之Kestrel07-277.asp.net core之配置07-278.asp.net core之Options07-319.asp.net core之日志08-0110.asp.net core之路由08-0211.asp.net core之异常处理08-0312.asp.net core之HttpClient08-0413.asp.net core之实时应用08-0414.asp.net core之EfCore08-0715.造轮子之自动依赖注入10-0816.造轮子之日志10-0817.造轮子之统一业务异常处理10-0818.造轮子之统一请求响应格式10-0819.造轮子之缓存10-0820.造轮子之ORM集成10-0821.造轮子之asp.net core identity10-0922.造轮子之自定义授权策略10-0923.造轮子之权限管理10-1024.造轮子之多语言管理10-1125.造轮子之角色管理10-1226.造轮子之用户管理10-1227.造轮子之菜单管理10-1228.造轮子之属性注入配合懒加载构建服务抽象基类10-1229.造轮子之EventBus10-1230.造轮子之消息实时推送10-1331.造轮子之种子数据10-1632.造轮子之集成GraphQL10-1633.造轮子之设置管理10-19

34.造轮子之文件管理10-23

收起

前面我们完成了设置管理，接下来正好配合设置管理来实现文件管理功能。
文件管理自然包括文件上传，下载以及文件存储功能。设计要求可以支持扩展多种存储服务，如本地文件，云存储等等。

数据库设计#

首先当然是我们的数据库表设计，用于管理文件。创建一个文件信息存储表。

using Wheel.Domain.Common;
using Wheel.Enums;

namespace Wheel.Domain.FileStorages
{
    /// 

    /// 文件信息存储表
    /// 
    public class FileStorage : Entity, IHasCreationTime
    {
        /// 

        /// 文件名
        /// 
        public string FileName { get; set; }
        /// 

        /// 文件类型ContentType
        /// 
        public string ContentType { get; set; }
        /// 

        /// 文件类型
        /// 
        public FileStorageType FileStorageType { get; set; }
        /// 

        /// 大小
        /// 
        public long Size { get; set; }
        /// 

        /// 存储路径
        /// 
        public string Path { get; set; }
        /// 

        /// 创建时间
        /// 
        public DateTimeOffset CreationTime { get; set; }
        /// 

        /// 存储类型
        /// 
        public string Provider { get; set; }
    }
}

namespace Wheel.Enums
{
    public enum FileStorageType
    {
        /// 

        /// 普通文件
        /// 
        File = 0,
        /// 

        /// 图片
        /// 
        Image = 1,
        /// 

        /// 视频
        /// 
        Video = 2,
        /// 

        /// 音频
        /// 
        Audio = 3,
        /// 

        /// 文本类型
        /// 
        Text = 4,
    }
}

FileStorageType是对ContentType类型的包装。后面可根据需求再加上细分类型。

using Wheel.Enums;

namespace Wheel.Domain.FileStorages
{
    public static class FileStorageTypeChecker
    {
        public static FileStorageType CheckFileType(string contentType)
        {
            return contentType switch
            {
                var _ when contentType.StartsWith("audio") => FileStorageType.Audio,
                var _ when contentType.StartsWith("image") => FileStorageType.Image,
                var _ when contentType.StartsWith("text") => FileStorageType.Text,
                var _ when contentType.StartsWith("video") => FileStorageType.Video,
                _ => FileStorageType.File
            };
        }
    }
}

Provider对应不同的存储服务。如Minio等。

修改DbContext#

在DbContext中添加代码：

#region FileStorage
public DbSet<FileStorage> FileStorages { get; set; }
#endregion


protected override void OnModelCreating(ModelBuilder builder)
{
    base.OnModelCreating(builder);

    ConfigureIdentity(builder);
    ConfigureLocalization(builder);
    ConfigurePermissionGrants(builder);
    ConfigureMenus(builder);
    ConfigureSettings(builder);
    ConfigureFileStorage(builder);
}

void ConfigureFileStorage(ModelBuilder builder)
{
    builder.Entity(b =>
    {
        b.HasKey(o => o.Id);
        b.Property(o => o.FileName).HasMaxLength(256);
        b.Property(o => o.Path).HasMaxLength(256);
        b.Property(o => o.ContentType).HasMaxLength(32);
        b.Property(o => o.Provider).HasMaxLength(32);
    });
}

然后执行数据库迁移操作即可完成表创建。

FileStorageProvider#

接下来就是实现我们的文件存储的Provider，首先创建一个IFileStorageProvider基础接口。

using Wheel.DependencyInjection;

namespace Wheel.FileStorages
{
    public interface IFileStorageProvider : ITransientDependency
    {
        string Name { get; }

        Task Upload(UploadFileArgs uploadFileArgs, CancellationToken cancellationToken = default);
        Task Download(DownloadFileArgs downloadFileArgs, CancellationToken cancellationToken = default);

        Task<object> GetClient();

        void ConfigureClient<T>(Action configure);

    }
}

提供定义名称，上传下载，以及获取Provider的Client和配置Provider中的Client的方法。

FileProviderSettingDefinition#

既然要对接各种存储服务，那么当然少不了对接的配置，那么我们就基于前面设置管理。添加一个FileProviderSettingDefinition

using Wheel.Enums;

namespace Wheel.Settings.FileProvider
{
    public class FileProviderSettingDefinition : ISettingDefinition
    {
        public string GroupName => "FileProvider";

        public SettingScope SettingScope => SettingScope.Global;

        public Dictionary<string, SettingValueParams> Define()
        {
            return new Dictionary<string, SettingValueParams>
            {
                { "Minio.Endpoint", new(SettingValueType.String, "127.0.0.1:9000") },
                { "Minio.AccessKey", new(SettingValueType.String, "2QgNxo11uxgULRvkrdaT") },
                { "Minio.SecretKey", new(SettingValueType.String, "NvzXnh81UMwEcvLJc8BslA1GA0j0sCq0aXRgHSRJ") },
                { "Minio.Region", new(SettingValueType.String) },
                { "Minio.SessionToken", new(SettingValueType.String) }
            };
        }
    }
}

这里我暂时只实现对接Minio，所以只加上Minio的配置。

MinioFileStorageProvider#

接下来实现一个MinioFileStorageProvider

using Minio;
using Minio.DataModel.Args;
using Minio.Exceptions;
using Wheel.Settings;

namespace Wheel.FileStorages.Providers
{
    public class MinioFileStorageProvider : IFileStorageProvider
    {
        private readonly ISettingProvider _settingProvider;
        private readonly ILogger _logger;

        public MinioFileStorageProvider(ISettingProvider settingProvider, ILogger logger)
        {
            _settingProvider = settingProvider;
            _logger = logger;
        }

        public string Name => "Minio";
        internal Action? Configure { get; private set; }
        public async Task Upload(UploadFileArgs uploadFileArgs, CancellationToken cancellationToken = default)
        {
            var client = await GetMinioClient();
            try
            {
                // Make a bucket on the server, if not already present.
                var beArgs = new BucketExistsArgs()
                    .WithBucket(uploadFileArgs.BucketName);
                bool found = await client.BucketExistsAsync(beArgs, cancellationToken).ConfigureAwait(false);
                if (!found)
                {
                    var mbArgs = new MakeBucketArgs()
                        .WithBucket(uploadFileArgs.BucketName);
                    await client.MakeBucketAsync(mbArgs, cancellationToken).ConfigureAwait(false);
                }
                // Upload a file to bucket.
                var putObjectArgs = new PutObjectArgs()
                    .WithBucket(uploadFileArgs.BucketName)
                    .WithObject(uploadFileArgs.FileName)
                    .WithStreamData(uploadFileArgs.FileStream)
                    .WithObjectSize(uploadFileArgs.FileStream.Length)
                    .WithContentType(uploadFileArgs.ContentType);
                await client.PutObjectAsync(putObjectArgs, cancellationToken).ConfigureAwait(false);
                var path = BuildPath(uploadFileArgs.BucketName, uploadFileArgs.FileName);
                _logger.LogInformation("Successfully Uploaded " + path);
                return new UploadFileResult { FilePath = path, Success = true };
            }
            catch (MinioException e)
            {
                _logger.LogError("File Upload Error: {0}", e.Message);
                return new UploadFileResult { Success = false };
            }
        }
        public async Task Download(DownloadFileArgs downloadFileArgs, CancellationToken cancellationToken = default)
        {
            var client = await GetMinioClient();
            try
            {
                var stream = new MemoryStream();
                var args = downloadFileArgs.Path.Split("/");
                var getObjectArgs = new GetObjectArgs()
                    .WithBucket(args[0])
                    .WithObject(downloadFileArgs.Path.RemovePreFix($"{args[0]}/"))
                    .WithCallbackStream(fs => fs.CopyTo(stream))
                    ;
                var response = await client.GetObjectAsync(getObjectArgs, cancellationToken).ConfigureAwait(false);

                _logger.LogInformation("Successfully Download " + downloadFileArgs.Path);
                stream.Position = 0;
                return new DownFileResult { Stream = stream, Success = true, FileName = response.ObjectName, ContentType = response.ContentType };
            }
            catch (MinioException e)
            {
                _logger.LogError("File Download Error: {0}", e.Message);
                return new DownFileResult { Success = false };
            }
        }

        public async Task<object> GetClient()
        {
            return await GetMinioClient();
        }

        public void ConfigureClient<T>(Action configure)
        {
            if (typeof(T) == typeof(IMinioClient))
                Configure = configure as Action;
            else
                throw new Exception("MinioFileProvider ConfigureClient Only Can Configure Type With IMinioClient");
        }

        private async Task GetMinioClient()
        {
            var minioSetting = await GetSettings();
            var client = new MinioClient()
                .WithHttpClient(new HttpClient())
                .WithEndpoint(minioSetting["Endpoint"])
                .WithCredentials(minioSetting["AccessKey"], minioSetting["SecretKey"])
                .WithSessionToken(minioSetting["SessionToken"]);

            if (!string.IsNullOrWhiteSpace(minioSetting["Region"]))
            {
                client.WithRegion(minioSetting["Region"]);
            }

            if (Configure != null)
            {
                Configure.Invoke(client);
            }
            return client;
        }

        private async Taskstring, string>> GetSettings()
        {
            var settings = await _settingProvider.GetGolbalSettings("FileProvider");

            return settings.Where(a => a.Key.StartsWith("Minio")).ToDictionary(a => a.Key.RemovePreFix("Minio."), a => a.Value);
        }
        private string BuildPath(string bucketName, string fileName)
        {
            return string.Join('/', bucketName, fileName);
        }
    }
}

这里定义MinioFileStorageProvider的Name是Minio用作标识。
Upload和Download则是正常的使用MinioClient的上传下载操作。
GetClient()返回一个MinioClient实例，用于方便做其他“骚操作”。
ConfigureClient则是用来配置MinioClient实例，代码约定限制只支持IMinioClient的类型。
GetSettings则是从SettingProvider中获取Minio的配置信息。

FileStorageManageAppService#

基础的对接搭好了，现在我们来实现我们的业务功能。很简单，就三个功能，上传下载，分页查询。

using Wheel.Core.Dto;
using Wheel.DependencyInjection;
using Wheel.Services.FileStorageManage.Dtos;

namespace Wheel.Services.FileStorageManage
{
    public interface IFileStorageManageAppService : ITransientDependency
    {
        Task> GetFileStoragePageList(FileStoragePageRequest request);
        Task>> UploadFiles(UploadFileDto uploadFileDto);
        Task> DownloadFile(long id);
    }
}

using Wheel.Const;
using Wheel.Core.Dto;
using Wheel.Core.Exceptions;
using Wheel.Domain;
using Wheel.Domain.FileStorages;
using Wheel.Enums;
using Wheel.FileStorages;
using Wheel.Services.FileStorageManage.Dtos;
using Path = System.IO.Path;

namespace Wheel.Services.FileStorageManage
{
    public class FileStorageManageAppService : WheelServiceBase, IFileStorageManageAppService
    {
        private readonly IBasicRepositorylong> _fileStorageRepository;

        public FileStorageManageAppService(IBasicRepositorylong> fileStorageRepository)
        {
            _fileStorageRepository = fileStorageRepository;
        }

        public async Task> GetFileStoragePageList(FileStoragePageRequest request)
        {
            var (items, total) = await _fileStorageRepository.GetPageListAsync(
                _fileStorageRepository.BuildPredicate(
                    (!string.IsNullOrWhiteSpace(request.FileName), f => f.FileName.Contains(request.FileName!)),
                    (!string.IsNullOrWhiteSpace(request.ContentType), f => f.ContentType.Equals(request.ContentType)),
                    (!string.IsNullOrWhiteSpace(request.Path), f => f.Path.StartsWith(request.Path!)),
                    (!string.IsNullOrWhiteSpace(request.Provider), f => f.Provider.Equals(request.Provider)),
                    (request.FileStorageType.HasValue, f => f.FileStorageType.Equals(request.FileStorageType))
                    ),
                (request.PageIndex -1) * request.PageSize,
                request.PageSize,
                request.OrderBy
                );

            return new Page(Mapper.Map>(items), total);
        }
        public async Task>> UploadFiles(UploadFileDto uploadFileDto)
        {
            var files = uploadFileDto.Files;
            if (files.Count == 0)
                return new R>(new());
            IFileStorageProvider? fileStorageProvider = null;
            var fileStorageProviders = ServiceProvider.GetServices();
            if (string.IsNullOrWhiteSpace(uploadFileDto.Provider))
            {
                fileStorageProvider = fileStorageProviders.First();
            }
            else
            {
                fileStorageProvider = fileStorageProviders.First(a => a.Name == uploadFileDto.Provider);
            }
            var fileStorages = new List();
            foreach (var file in files) 
            {
                var fileName = uploadFileDto.Cover ? file.FileName : $"{Path.GetFileNameWithoutExtension(file.FileName)}-{SnowflakeIdGenerator.Create()}{Path.GetExtension(file.FileName)}";
                var fileStream = file.OpenReadStream();
                var fileStorageType = FileStorageTypeChecker.CheckFileType(file.ContentType);
                var uploadFileArgs = new UploadFileArgs 
                {
                    BucketName = fileStorageType switch
                    {
                        FileStorageType.Image => "images",
                        FileStorageType.Video => "videos",
                        FileStorageType.Audio => "audios",
                        FileStorageType.Text => "texts",
                        _ => "files"
                    },
                    ContentType = file.ContentType,
                    FileName = fileName,
                    FileStream = fileStream
                };
                var uploadFileResult = await fileStorageProvider.Upload(uploadFileArgs);

                if (uploadFileResult.Success)
                {
                    var fileStorage = await _fileStorageRepository.InsertAsync(new FileStorage 
                    {
                        Id = SnowflakeIdGenerator.Create(),
                        ContentType = file.ContentType,
                        FileName = file.FileName,
                        FileStorageType = fileStorageType,
                        Path = uploadFileResult.FilePath,
                        Provider = fileStorageProvider.Name,
                        Size = fileStream.Length
                    });
                    await _fileStorageRepository.SaveChangeAsync();
                    fileStorages.Add(fileStorage);
                }
            }
            return new R>(Mapper.Map>(fileStorages));
        }

        public async Task> DownloadFile(long id)
        {
            var fileStorage = await _fileStorageRepository.FindAsync(id);
            if(fileStorage == null) 
            {
                throw new BusinessException(ErrorCode.FileNotExist, "FileNotExist")
                    .WithMessageDataData(id.ToString());
            }
            var fileStorageProvider = ServiceProvider.GetServices().First(a=>a.Name == fileStorage.Provider);

            var downloadResult = await fileStorageProvider.Download(new DownloadFileArgs { Path = fileStorage.Path });
            if (downloadResult.Success)
            {
                return new R(new DownloadFileResonse { ContentType = downloadResult.ContentType, FileName = downloadResult.FileName, Stream = downloadResult.Stream });
            }
            else
            {
                throw new BusinessException(ErrorCode.FileDownloadFail, "FileDownloadFail")
                    .WithMessageDataData(id.ToString());
            }
        }
    }
}

UploadFiles时如果没有指定Provider则默认取依赖注入第一个Provider，如果指定则取Provider。

using Microsoft.AspNetCore.Mvc;

namespace Wheel.Services.FileStorageManage.Dtos
{
    public class UploadFileDto
    {
        [FromQuery]
        public bool Cover { get; set; } = false;

        [FromQuery]
        public string? Provider { get; set; }

        [FromForm]
        public IFormFileCollection Files { get; set; }
    }
}

这里上传参数定义，Cover表示是否覆盖原文件，Provider表示指定那种存储服务。Files则是从Form表单中读取文件流。

FileController#

接下来就是把Service包成API对外。

using Microsoft.AspNetCore.Mvc;
using Wheel.Core.Dto;
using Wheel.Services.FileStorageManage;
using Wheel.Services.FileStorageManage.Dtos;

namespace Wheel.Controllers
{
    /// 

    /// 文件管理
    /// 
    [Route("api/[controller]")]
    [ApiController]
    public class FileController : WheelControllerBase
    {
        private readonly IFileStorageManageAppService _fileStorageManageAppService;

        public FileController(IFileStorageManageAppService fileStorageManageAppService)
        {
            _fileStorageManageAppService = fileStorageManageAppService;
        }
        /// 

        /// 分页查询列表
        /// 
        /// 
        /// 
        [HttpGet]
        public Task> GetFileStoragePageList([FromQuery] FileStoragePageRequest request)
        {
            return _fileStorageManageAppService.GetFileStoragePageList(request);
        }
        /// 

        /// 上传文件
        /// 
        /// 
        /// 
        [HttpPost]
        public Task>> UploadFiles(UploadFileDto uploadFileDto)
        {
            return _fileStorageManageAppService.UploadFiles(uploadFileDto);
        }
        /// 

        /// 下载文件
        /// 
        /// 
        /// 
        [HttpGet("{id}")]
        public async Task DownloadFile(long id)
        {
            var result = await _fileStorageManageAppService.DownloadFile(id);
            return File(result.Data.Stream, result.Data.ContentType, result.Data.FileName);
        }
    }
}

DownloadFile返回一个FileResult，浏览器会自动下载。

测试#

这里我使用本地的Minio服务进行测试。
查询

上传

可以看到我们FileName和Path不一样，默认不覆盖的情况，所有文件在后面自动拼接雪花Id。
下载文件

这里swagger可以看到有个Download file，点击即可下载出来

测试顺利完成，到这我们就完成了我们简单的文件管理功能了。

轮子仓库地址https://github.com/Wheel-Framework/Wheel
欢迎进群催更。