• 机器学习周记（第三十八周：语义分割）2024.5.6~2024.5.12

    目录

    摘要

    ABSTRACT

    1 DeeplabV3+实现思路

    预测部分

    ①主干网络介绍​编辑

    ② 加强特征提取结构

    ③ 利用特征获得预测结果

    摘要

      本周继续了语义分割的学习，主要学习了DeepLabV3+的部分实现思路，即DeepLabV3+的整个模型的预测过程，并通过代码进行了实现。待预测图片首先通过主干网络（MobilenetV2）进行特征提取，得到两个有效特征层，一个有效特征层是输入图片高和宽压缩两次的结果，一个有效特征层是输入图片高和宽压缩四次的结果。然后再通过加强特征提取，最后输出预测结果。

    ABSTRACT

    This week, We continued learning about semantic segmentation, focusing on the implementation details of DeepLabV3+. Specifically, I studied the prediction process of the entire DeepLabV3+ model and implemented it through code. For the prediction, the input image is first passed through the backbone network (MobileNetV2) for feature extraction, resulting in two effective feature layers. One effective feature layer is obtained by compressing the height and width of the input image twice, while the other is obtained by compressing them four times. Then, enhanced feature extraction is performed, followed by the final output of the prediction results.

    1 DeeplabV3+实现思路

    预测部分

    ①主干网络介绍

    DeeplabV3+在论文中采用的是Xception系列作为主干特征提取网络，除了Xception主干网络外，还可以使用ResNet和mobilenetv2等。这里主要使用mobilenetv2。

    MobileNet模型是Google针对手机等嵌入式设备提出的一种轻量级的深层神经网络。

    MobileNetV2是MobileNet的升级版，它具有一个非常重要的特点就是使用了Inverted resblock，整个mobilenetv2都由Inverted resblock组成。

    Inverted resblock可以分为两个部分：
    左边是主干部分，首先利用1x1卷积进行升维，然后利用3x3深度可分离卷积进行特征提取，然后再利用1x1卷积降维。
    右边是残差边部分，输入和输出直接相接。

    需要注意的是，在DeeplabV3当中，一般不会5次下采样，可选的有3次下采样和4次下采样，这里使用的4次下采样。这里所提到的下采样指的是不会进行五次长和宽的压缩，通常选用三次或者四次长和宽的压缩。

    在完成MobilenetV2的特征提取后，我们可以获得两个有效特征层，一个有效特征层是输入图片高和宽压缩两次的结果，一个有效特征层是输入图片高和宽压缩四次的结果。

    1. import math
    2. import os
    3.  
    4. import torch
    5. import torch.nn as nn
    6. import torch.utils.model_zoo as model_zoo
    7.  
    8. BatchNorm2d = nn.BatchNorm2d
    9.  
    10. def conv_bn(inp, oup, stride):
    11.     return nn.Sequential(
    12.         nn.Conv2d(inp, oup, 3, stride, 1, bias=False),
    13.         BatchNorm2d(oup),
    14.         nn.ReLU6(inplace=True)
    15.     )
    16.  
    17. def conv_1x1_bn(inp, oup):
    18.     return nn.Sequential(
    19.         nn.Conv2d(inp, oup, 1, 1, 0, bias=False),
    20.         BatchNorm2d(oup),
    21.         nn.ReLU6(inplace=True)
    22.     )
    23.  
    24. class InvertedResidual(nn.Module):
    25.     def __init__(self, inp, oup, stride, expand_ratio):
    26.         super(InvertedResidual, self).__init__()
    27.         self.stride = stride
    28.         assert stride in [1, 2]
    29.  
    30.         hidden_dim = round(inp * expand_ratio)
    31.         self.use_res_connect = self.stride == 1 and inp == oup
    32.  
    33.         if expand_ratio == 1:
    34.             self.conv = nn.Sequential(
    35.                 # dw
    36.                 nn.Conv2d(hidden_dim, hidden_dim, 3, stride, 1, groups=hidden_dim, bias=False),
    37.                 BatchNorm2d(hidden_dim),
    38.                 nn.ReLU6(inplace=True),
    39.                 # pw-linear
    40.                 nn.Conv2d(hidden_dim, oup, 1, 1, 0, bias=False),
    41.                 BatchNorm2d(oup),
    42.             )
    43.         else:
    44.             self.conv = nn.Sequential(
    45.                 # pw
    46.                 nn.Conv2d(inp, hidden_dim, 1, 1, 0, bias=False),
    47.                 BatchNorm2d(hidden_dim),
    48.                 nn.ReLU6(inplace=True),
    49.                 # dw
    50.                 nn.Conv2d(hidden_dim, hidden_dim, 3, stride, 1, groups=hidden_dim, bias=False),
    51.                 BatchNorm2d(hidden_dim),
    52.                 nn.ReLU6(inplace=True),
    53.                 # pw-linear
    54.                 nn.Conv2d(hidden_dim, oup, 1, 1, 0, bias=False),
    55.                 BatchNorm2d(oup),
    56.             )
    57.  
    58.     def forward(self, x):
    59.         if self.use_res_connect:
    60.             return x + self.conv(x)
    61.         else:
    62.             return self.conv(x)
    63.  
    64. class MobileNetV2(nn.Module):
    65.     def __init__(self, n_class=1000, input_size=224, width_mult=1.):
    66.         super(MobileNetV2, self).__init__()
    67.         block = InvertedResidual
    68.         input_channel = 32
    69.         last_channel = 1280
    70.         interverted_residual_setting = [
    71.             # t, c, n, s
    72.             [1, 16, 1, 1],
    73.             [6, 24, 2, 2],
    74.             [6, 32, 3, 2],
    75.             [6, 64, 4, 2],
    76.             [6, 96, 3, 1],
    77.             [6, 160, 3, 2],
    78.             [6, 320, 1, 1],
    79.         ]
    80.  
    81.         # building first layer
    82.         assert input_size % 32 == 0
    83.         input_channel = int(input_channel * width_mult)
    84.         self.last_channel = int(last_channel * width_mult) if width_mult > 1.0 else last_channel
    85.         self.features = [conv_bn(3, input_channel, 2)]
    86.         # building inverted residual blocks
    87.         for t, c, n, s in interverted_residual_setting:
    88.             output_channel = int(c * width_mult)
    89.             for i in range(n):
    90.                 if i == 0:
    91.                     self.features.append(block(input_channel, output_channel, s, expand_ratio=t))
    92.                 else:
    93.                     self.features.append(block(input_channel, output_channel, 1, expand_ratio=t))
    94.                 input_channel = output_channel
    95.         # building last several layers
    96.         self.features.append(conv_1x1_bn(input_channel, self.last_channel))
    97.         # make it nn.Sequential
    98.         self.features = nn.Sequential(*self.features)
    99.  
    100.         # building classifier
    101.         self.classifier = nn.Sequential(
    102.             nn.Dropout(0.2),
    103.             nn.Linear(self.last_channel, n_class),
    104.         )
    105.  
    106.         self._initialize_weights()
    107.  
    108.     def forward(self, x):
    109.         x = self.features(x)
    110.         x = x.mean(3).mean(2)
    111.         x = self.classifier(x)
    112.         return x
    113.  
    114.     def _initialize_weights(self):
    115.         for m in self.modules():
    116.             if isinstance(m, nn.Conv2d):
    117.                 n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
    118.                 m.weight.data.normal_(0, math.sqrt(2. / n))
    119.                 if m.bias is not None:
    120.                     m.bias.data.zero_()
    121.             elif isinstance(m, BatchNorm2d):
    122.                 m.weight.data.fill_(1)
    123.                 m.bias.data.zero_()
    124.             elif isinstance(m, nn.Linear):
    125.                 n = m.weight.size(1)
    126.                 m.weight.data.normal_(0, 0.01)
    127.                 m.bias.data.zero_()
    128.  
    129.  
    130. def load_url(url, model_dir='./model_data', map_location=None):
    131.     if not os.path.exists(model_dir):
    132.         os.makedirs(model_dir)
    133.     filename = url.split('/')[-1]
    134.     cached_file = os.path.join(model_dir, filename)
    135.     if os.path.exists(cached_file):
    136.         return torch.load(cached_file, map_location=map_location)
    137.     else:
    138.         return model_zoo.load_url(url,model_dir=model_dir)
    139.  
    140. def mobilenetv2(pretrained=False, **kwargs):
    141.     model = MobileNetV2(n_class=1000, **kwargs)
    142.     if pretrained:
    143.         model.load_state_dict(load_url('http://sceneparsing.csail.mit.edu/model/pretrained_resnet/mobilenet_v2.pth.tar'), strict=False)
    144.     return model
    ② 加强特征提取结构

    在DeeplabV3+中，加强特征提取网络可以分为两部分：
    在Encoder中，我们会对压缩四次的初步有效特征层利用并行的Atrous Convolution，分别用不同rate的Atrous Convolution进行特征提取，再进行合并，再进行1x1卷积压缩特征。

    在Decoder中，我们会对压缩两次的初步有效特征层利用1x1卷积调整通道数，再和空洞卷积后的有效特征层上采样的结果进行堆叠，在完成堆叠后，进行两次深度可分离卷积块。
     

    在这里插入图片描述

    这个时候，我们就获得了一个最终的有效特征层，它是整张图片的特征浓缩。

    1. import torch
    2. import torch.nn as nn
    3. import torch.nn.functional as F
    4. from nets.xception import xception
    5. from nets.mobilenetv2 import mobilenetv2
    6.  
    7. class MobileNetV2(nn.Module):
    8.     def __init__(self, downsample_factor=8, pretrained=True):
    9.         super(MobileNetV2, self).__init__()
    10.         from functools import partial
    11.         
    12.         model = mobilenetv2(pretrained)
    13.         self.features = model.features[:-1]
    14.  
    15.         self.total_idx = len(self.features)
    16.         self.down_idx = [2, 4, 7, 14]
    17.  
    18.         if downsample_factor == 8:
    19.             for i in range(self.down_idx[-2], self.down_idx[-1]):
    20.                 self.features[i].apply(
    21.                     partial(self._nostride_dilate, dilate=2)
    22.                 )
    23.             for i in range(self.down_idx[-1], self.total_idx):
    24.                 self.features[i].apply(
    25.                     partial(self._nostride_dilate, dilate=4)
    26.                 )
    27.         elif downsample_factor == 16:
    28.             for i in range(self.down_idx[-1], self.total_idx):
    29.                 self.features[i].apply(
    30.                     partial(self._nostride_dilate, dilate=2)
    31.                 )
    32.         
    33.     def _nostride_dilate(self, m, dilate):
    34.         classname = m.__class__.__name__
    35.         if classname.find('Conv') != -1:
    36.             if m.stride == (2, 2):
    37.                 m.stride = (1, 1)
    38.                 if m.kernel_size == (3, 3):
    39.                     m.dilation = (dilate//2, dilate//2)
    40.                     m.padding = (dilate//2, dilate//2)
    41.             else:
    42.                 if m.kernel_size == (3, 3):
    43.                     m.dilation = (dilate, dilate)
    44.                     m.padding = (dilate, dilate)
    45.  
    46.     def forward(self, x):
    47.         low_level_features = self.features[:4](x)
    48.         x = self.features[4:](low_level_features)
    49.         return low_level_features, x 
    50.  
    51.  
    52. #-----------------------------------------#
    53. #   ASPP特征提取模块
    54. #   利用不同膨胀率的膨胀卷积进行特征提取
    55. #-----------------------------------------#
    56. class ASPP(nn.Module):
    57. 	def __init__(self, dim_in, dim_out, rate=1, bn_mom=0.1):
    58. 		super(ASPP, self).__init__()
    59. 		self.branch1 = nn.Sequential(
    60. 				nn.Conv2d(dim_in, dim_out, 1, 1, padding=0, dilation=rate,bias=True),
    61. 				nn.BatchNorm2d(dim_out, momentum=bn_mom),
    62. 				nn.ReLU(inplace=True),
    63. 		)
    64. 		self.branch2 = nn.Sequential(
    65. 				nn.Conv2d(dim_in, dim_out, 3, 1, padding=6*rate, dilation=6*rate, bias=True),
    66. 				nn.BatchNorm2d(dim_out, momentum=bn_mom),
    67. 				nn.ReLU(inplace=True),	
    68. 		)
    69. 		self.branch3 = nn.Sequential(
    70. 				nn.Conv2d(dim_in, dim_out, 3, 1, padding=12*rate, dilation=12*rate, bias=True),
    71. 				nn.BatchNorm2d(dim_out, momentum=bn_mom),
    72. 				nn.ReLU(inplace=True),	
    73. 		)
    74. 		self.branch4 = nn.Sequential(
    75. 				nn.Conv2d(dim_in, dim_out, 3, 1, padding=18*rate, dilation=18*rate, bias=True),
    76. 				nn.BatchNorm2d(dim_out, momentum=bn_mom),
    77. 				nn.ReLU(inplace=True),	
    78. 		)
    79. 		self.branch5_conv = nn.Conv2d(dim_in, dim_out, 1, 1, 0,bias=True)
    80. 		self.branch5_bn = nn.BatchNorm2d(dim_out, momentum=bn_mom)
    81. 		self.branch5_relu = nn.ReLU(inplace=True)
    82.  
    83. 		self.conv_cat = nn.Sequential(
    84. 				nn.Conv2d(dim_out*5, dim_out, 1, 1, padding=0,bias=True),
    85. 				nn.BatchNorm2d(dim_out, momentum=bn_mom),
    86. 				nn.ReLU(inplace=True),		
    87. 		)
    88.  
    89. 	def forward(self, x):
    90. 		[b, c, row, col] = x.size()
    91.         #-----------------------------------------#
    92.         #   一共五个分支
    93.         #-----------------------------------------#
    94. 		conv1x1 = self.branch1(x)
    95. 		conv3x3_1 = self.branch2(x)
    96. 		conv3x3_2 = self.branch3(x)
    97. 		conv3x3_3 = self.branch4(x)
    98.         #-----------------------------------------#
    99.         #   第五个分支，全局平均池化+卷积
    100.         #-----------------------------------------#
    101. 		global_feature = torch.mean(x,2,True)
    102. 		global_feature = torch.mean(global_feature,3,True)
    103. 		global_feature = self.branch5_conv(global_feature)
    104. 		global_feature = self.branch5_bn(global_feature)
    105. 		global_feature = self.branch5_relu(global_feature)
    106. 		global_feature = F.interpolate(global_feature, (row, col), None, 'bilinear', True)
    107. 		
    108.         #-----------------------------------------#
    109.         #   将五个分支的内容堆叠起来
    110.         #   然后1x1卷积整合特征。
    111.         #-----------------------------------------#
    112. 		feature_cat = torch.cat([conv1x1, conv3x3_1, conv3x3_2, conv3x3_3, global_feature], dim=1)
    113. 		result = self.conv_cat(feature_cat)
    114. 		return result
    115.  
    116. class DeepLab(nn.Module):
    117.     def __init__(self, num_classes, backbone="mobilenet", pretrained=True, downsample_factor=16):
    118.         super(DeepLab, self).__init__()
    119.         if backbone=="xception":
    120.             #----------------------------------#
    121.             #   获得两个特征层
    122.             #   浅层特征    [128,128,256]
    123.             #   主干部分    [30,30,2048]
    124.             #----------------------------------#
    125.             self.backbone = xception(downsample_factor=downsample_factor, pretrained=pretrained)
    126.             in_channels = 2048
    127.             low_level_channels = 256
    128.         elif backbone=="mobilenet":
    129.             #----------------------------------#
    130.             #   获得两个特征层
    131.             #   浅层特征    [128,128,24]
    132.             #   主干部分    [30,30,320]
    133.             #----------------------------------#
    134.             self.backbone = MobileNetV2(downsample_factor=downsample_factor, pretrained=pretrained)
    135.             in_channels = 320
    136.             low_level_channels = 24
    137.         else:
    138.             raise ValueError('Unsupported backbone - `{}`, Use mobilenet, xception.'.format(backbone))
    139.  
    140.         #-----------------------------------------#
    141.         #   ASPP特征提取模块
    142.         #   利用不同膨胀率的膨胀卷积进行特征提取
    143.         #-----------------------------------------#
    144.         self.aspp = ASPP(dim_in=in_channels, dim_out=256, rate=16//downsample_factor)
    145.         
    146.         #----------------------------------#
    147.         #   浅层特征边
    148.         #----------------------------------#
    149.         self.shortcut_conv = nn.Sequential(
    150.             nn.Conv2d(low_level_channels, 48, 1),
    151.             nn.BatchNorm2d(48),
    152.             nn.ReLU(inplace=True)
    153.         )		
    154.  
    155.         self.cat_conv = nn.Sequential(
    156.             nn.Conv2d(48+256, 256, 3, stride=1, padding=1),
    157.             nn.BatchNorm2d(256),
    158.             nn.ReLU(inplace=True),
    159.             nn.Dropout(0.5),
    160.  
    161.             nn.Conv2d(256, 256, 3, stride=1, padding=1),
    162.             nn.BatchNorm2d(256),
    163.             nn.ReLU(inplace=True),
    164.  
    165.             nn.Dropout(0.1),
    166.         )
    167.         self.cls_conv = nn.Conv2d(256, num_classes, 1, stride=1)
    168.  
    169.     def forward(self, x):
    170.         H, W = x.size(2), x.size(3)
    171.         #-----------------------------------------#
    172.         #   获得两个特征层
    173.         #   浅层特征-进行卷积处理
    174.         #   主干部分-利用ASPP结构进行加强特征提取
    175.         #-----------------------------------------#
    176.         low_level_features, x = self.backbone(x)
    177.         x = self.aspp(x)
    178.         low_level_features = self.shortcut_conv(low_level_features)
    179.         
    180.         #-----------------------------------------#
    181.         #   将加强特征边上采样
    182.         #   与浅层特征堆叠后利用卷积进行特征提取
    183.         #-----------------------------------------#
    184.         x = F.interpolate(x, size=(low_level_features.size(2), low_level_features.size(3)), mode='bilinear', align_corners=True)
    185.         x = self.cat_conv(torch.cat((x, low_level_features), dim=1))
    186.         x = self.cls_conv(x)
    187.         x = F.interpolate(x, size=(H, W), mode='bilinear', align_corners=True)
    188.         return x
    ③ 利用特征获得预测结果

    利用1、2步，我们可以获取输入进来的图片的特征，此时，我们需要利用特征获得预测结果。

    利用特征获得预测结果的过程可以分为2步：
    1、利用一个1x1卷积进行通道调整，调整成Num_Classes。
    2、利用resize进行上采样使得最终输出层，宽高和输入图片一样。

    1. class DeepLab(nn.Module):
    2.     def __init__(self, num_classes, backbone="mobilenet", pretrained=True, downsample_factor=16):
    3.         super(DeepLab, self).__init__()
    4.         if backbone=="xception":
    5.             #----------------------------------#
    6.             #   获得两个特征层
    7.             #   浅层特征    [128,128,256]
    8.             #   主干部分    [30,30,2048]
    9.             #----------------------------------#
    10.             self.backbone = xception(downsample_factor=downsample_factor, pretrained=pretrained)
    11.             in_channels = 2048
    12.             low_level_channels = 256
    13.         elif backbone=="mobilenet":
    14.             #----------------------------------#
    15.             #   获得两个特征层
    16.             #   浅层特征    [128,128,24]
    17.             #   主干部分    [30,30,320]
    18.             #----------------------------------#
    19.             self.backbone = MobileNetV2(downsample_factor=downsample_factor, pretrained=pretrained)
    20.             in_channels = 320
    21.             low_level_channels = 24
    22.         else:
    23.             raise ValueError('Unsupported backbone - `{}`, Use mobilenet, xception.'.format(backbone))
    24.  
    25.         #-----------------------------------------#
    26.         #   ASPP特征提取模块
    27.         #   利用不同膨胀率的膨胀卷积进行特征提取
    28.         #-----------------------------------------#
    29.         self.aspp = ASPP(dim_in=in_channels, dim_out=256, rate=16//downsample_factor)
    30.         
    31.         #----------------------------------#
    32.         #   浅层特征边
    33.         #----------------------------------#
    34.         self.shortcut_conv = nn.Sequential(
    35.             nn.Conv2d(low_level_channels, 48, 1),
    36.             nn.BatchNorm2d(48),
    37.             nn.ReLU(inplace=True)
    38.         )		
    39.  
    40.         self.cat_conv = nn.Sequential(
    41.             nn.Conv2d(48+256, 256, 3, stride=1, padding=1),
    42.             nn.BatchNorm2d(256),
    43.             nn.ReLU(inplace=True),
    44.             nn.Dropout(0.5),
    45.  
    46.             nn.Conv2d(256, 256, 3, stride=1, padding=1),
    47.             nn.BatchNorm2d(256),
    48.             nn.ReLU(inplace=True),
    49.  
    50.             nn.Dropout(0.1),
    51.         )
    52.         self.cls_conv = nn.Conv2d(256, num_classes, 1, stride=1)
    53.  
    54.     def forward(self, x):
    55.         H, W = x.size(2), x.size(3)
    56.         #-----------------------------------------#
    57.         #   获得两个特征层
    58.         #   浅层特征-进行卷积处理
    59.         #   主干部分-利用ASPP结构进行加强特征提取
    60.         #-----------------------------------------#
    61.         low_level_features, x = self.backbone(x)
    62.         x = self.aspp(x)
    63.         low_level_features = self.shortcut_conv(low_level_features)
    64.         
    65.         #-----------------------------------------#
    66.         #   将加强特征边上采样
    67.         #   与浅层特征堆叠后利用卷积进行特征提取
    68.         #-----------------------------------------#
    69.         x = F.interpolate(x, size=(low_level_features.size(2), low_level_features.size(3)), mode='bilinear', align_corners=True)
    70.         x = self.cat_conv(torch.cat((x, low_level_features), dim=1))
    71.         x = self.cls_conv(x)
    72.         x = F.interpolate(x, size=(H, W), mode='bilinear', align_corners=True)
    73.         return x
  • 相关阅读:
    网站死链检测的软件-网站死链检测的工具
    交换机与路由技术-15-链路聚合
    【多线程】常见的锁策略
    2020年中职组“网络安全”赛项广东省竞赛任务书
    ByteArray转byte[]的两种方式
    阿里云新用户：定义，专享福利及优惠活动
    [MAUI程序设计]界面多态与实现
    sprigboot+在线预定车位管理 毕业设计-附源码221738
    Java毕业设计-在线点餐系统
    HTB靶场之OnlyForYou
  • 原文地址：https://blog.csdn.net/DominaterWE/article/details/138746877