• 【实例分割】论文详解YOLACT：Real-time Instance Segmentation

    🏆论文下载：paper

    🏆代码下载：code

    目录

    🏆论文下载：paper

    🏆代码下载：code

    1.🌷🌷创新点

    2.🌷🌷网络结构

    2.1🍀🍀Backbone

    2.2🍀🍀Protonet

    2.3🍀🍀Prediction Head

    2.4🍀🍀Masks Assembly

    3.🌷🌷结果

    3.1🍀🍀prototypes表现

    3.2🍀🍀coco结果

    4.🌷🌷代码

    整理不易，欢迎一键三连！！！

    送你们一条美丽的--分割线--

    YOLACT是比较经典的one-stage实例分割方法，属于anchor-base流派，即需要anchor计算。

    YOLACT：You Only Look At CoefficienTs（系数复数）

    摘要：我们提出了一个简单的用于实时实例分割的全卷积模型，在 MS COCO上实现了 29.8 mAP，在单个 Titan Xp 上达到 33.5 fps ，比以前任何竞争方法都要快得多。
    此外，我们仅在一个训练后就得到了这个结果图形处理器。 我们通过将实例分割分成两个并行子任务来实现这一点：（1）生成一组原型掩码和（2）预测每个实例掩码系数。 然后，我们通过将原型与掩码系数线性组合来生成实例掩码。 我们发现因为这个过程不依赖于重新池化，所以这种方法可以产生非常高质量的掩模，并且免费表现出时间稳定性。 此外，我们分析了模型的出色表现，虽然是全卷积的，但是它们学会以翻译变体方式自行本地化实例。 最后，我们还提出了快速 NMS，它是标准 NMS 的快 12 毫秒的替代品，但仅有边际性能损失。

    1.🌷🌷创新点

    • 针对实例分割任务提出了一种简单的全卷积模型；

    • 这个模型在COCO数据集用一块Titan Xp完成了29.8mAP和33.5fps的实时分割效果；

    • 将任务分为两个平行的子任务。产生prototype masks和预测mask coefficients；

      • prototype masks
        卷积层：在提取空间相关信息上效果显著。

      • mask coefficients
        全连接层：在获取语义向量上效果显著。

    • 利用矩阵计算对NMS进行加速

    2.🌷🌷网络结构

    左侧(prototypes)：蓝（低值）黄（高值）

    中间(NMS/CROP/THRESHOLD)：灰（没有训练的函数）

    网络结构：RetinaNet ResNet-101+FPN

    2.1🍀🍀Backbone

            采用卷积神经网络（例如ResNet101）做特征提取，然后将提取到的特征输入到特征金字塔网络中进行不同层级特征提取。ResNet的网络结构如图所示。从图中可以看出，ResNet的卷积模块一共有5个从conv1,conv2_x到conv5_x，分别对应图1 YOLACT模型中的C1-C5，然后将提取到的特征输入到特征金字塔网络中。YOLACT采用了多尺度的特征图， 从而可以检测到不同尺寸的物体，也就是在大的特征图上检测小的物体，在小的特征图上检测大的物体。

    * 特征金字塔模块的主要作用是获取深度更深的特征图，且含有多个不同尺度的特征图。

    2.2🍀🍀Protonet

            Protonet模块通过卷积和上采样获得Prototypes，Prototypes是多张mask的，mask中的亮（值大）的区域就是目标区域。最终通过线性组合生成的mask来获取每个实例的mask。

    2.3🍀🍀Prediction Head

            预测头在RetinaNet的基础上多回归了一个mask系数，输出预测框Bbox，类别信息conf以及掩码系数。利用此系数与Protonet中的mask线性组合。YOLACT利用特征金字塔中的特征图（5个尺度）每个特征图的每个点都生成3个目标框，为了避免太多冗余目标框，使用NMS非极大值抑制来进行目标框筛选。

    2.4🍀🍀Masks Assembly

            利用Prediction Head模块的mask系数与Protonet中的多张mask进行线性组合，每个目标得到一张mask。

    损失函数：Lmask = BCE(M, Mgt)

    3.🌷🌷结果

    3.1🍀🍀prototypes表现

    3.2🍀🍀coco结果

    4.🌷🌷代码

            YOLACT类的定义：

    1. class Yolact(nn.Module):
    2.     """
    3.     You can set the arguments by changing them in the backbone config object in config.py.
    5.     Parameters (in cfg.backbone):
    6.         - selected_layers: The indices of the conv layers to use for prediction.
    7.         - pred_scales:     A list with len(selected_layers) containing tuples of scales (see PredictionModule)
    8.         - pred_aspect_ratios: A list of lists of aspect ratios with len(selected_layers) (see PredictionModule)
    9.     """
    10.  
    11.     def __init__(self):
    12.         super().__init__()
    13.  
    14.         self.backbone = construct_backbone(cfg.backbone)
    15.  
    16.         if cfg.freeze_bn:
    17.             self.freeze_bn()
    18.  
    19.         # Compute mask_dim here and add it back to the config. Make sure Yolact's constructor is called early!
    20.         if cfg.mask_type == mask_type.direct:
    21.             cfg.mask_dim = cfg.mask_size**2
    22.         elif cfg.mask_type == mask_type.lincomb:
    23.             if cfg.mask_proto_use_grid:
    24.                 self.grid = torch.Tensor(np.load(cfg.mask_proto_grid_file))
    25.                 self.num_grids = self.grid.size(0)
    26.             else:
    27.                 self.num_grids = 0
    28.  
    29.             self.proto_src = cfg.mask_proto_src
    30.             
    31.             if self.proto_src is None: in_channels = 3
    32.             elif cfg.fpn is not None: in_channels = cfg.fpn.num_features
    33.             else: in_channels = self.backbone.channels[self.proto_src]
    34.             in_channels += self.num_grids
    35.  
    36.             # The include_last_relu=false here is because we might want to change it to another function
    37.             self.proto_net, cfg.mask_dim = make_net(in_channels, cfg.mask_proto_net, include_last_relu=False)
    38.  
    39.             if cfg.mask_proto_bias:
    40.                 cfg.mask_dim += 1
    41.  
    42.  
    43.         self.selected_layers = cfg.backbone.selected_layers
    44.         src_channels = self.backbone.channels
    45.  
    46.         if cfg.use_maskiou:
    47.             self.maskiou_net = FastMaskIoUNet()
    48.  
    49.         if cfg.fpn is not None:
    50.             # Some hacky rewiring to accomodate the FPN
    51.             self.fpn = FPN([src_channels[i] for i in self.selected_layers])
    52.             self.selected_layers = list(range(len(self.selected_layers) + cfg.fpn.num_downsample))
    53.             src_channels = [cfg.fpn.num_features] * len(self.selected_layers)
    54.  
    55.  
    56.         self.prediction_layers = nn.ModuleList()
    57.         cfg.num_heads = len(self.selected_layers)
    58.  
    59.         for idx, layer_idx in enumerate(self.selected_layers):
    60.             # If we're sharing prediction module weights, have every module's parent be the first one
    61.             parent = None
    62.             if cfg.share_prediction_module and idx > 0:
    63.                 parent = self.prediction_layers[0]
    64.  
    65.             pred = PredictionModule(src_channels[layer_idx], src_channels[layer_idx],
    66.                                     aspect_ratios = cfg.backbone.pred_aspect_ratios[idx],
    67.                                     scales        = cfg.backbone.pred_scales[idx],
    68.                                     parent        = parent,
    69.                                     index         = idx)
    70.             self.prediction_layers.append(pred)
    71.  
    72.         # Extra parameters for the extra losses
    73.         if cfg.use_class_existence_loss:
    74.             # This comes from the smallest layer selected
    75.             # Also note that cfg.num_classes includes background
    76.             self.class_existence_fc = nn.Linear(src_channels[-1], cfg.num_classes - 1)
    77.         
    78.         if cfg.use_semantic_segmentation_loss:
    79.             self.semantic_seg_conv = nn.Conv2d(src_channels[0], cfg.num_classes-1, kernel_size=1)
    80.  
    81.         # For use in evaluation
    82.         self.detect = Detect(cfg.num_classes, bkg_label=0, top_k=cfg.nms_top_k,
    83.             conf_thresh=cfg.nms_conf_thresh, nms_thresh=cfg.nms_thresh)
    84.  
    85.     def save_weights(self, path):
    86.         """ Saves the model's weights using compression because the file sizes were getting too big. """
    87.         torch.save(self.state_dict(), path)
    88.     
    89.     def load_weights(self, path):
    90.         """ Loads weights from a compressed save file. """
    91.         state_dict = torch.load(path)
    92.  
    93.         # For backward compatability, remove these (the new variable is called layers)
    94.         for key in list(state_dict.keys()):
    95.             if key.startswith('backbone.layer') and not key.startswith('backbone.layers'):
    96.                 del state_dict[key]
    97.         
    98.             # Also for backward compatibility with v1.0 weights, do this check
    99.             if key.startswith('fpn.downsample_layers.'):
    100.                 if cfg.fpn is not None and int(key.split('.')[2]) >= cfg.fpn.num_downsample:
    101.                     del state_dict[key]
    102.         self.load_state_dict(state_dict)
    103.  
    104.     def init_weights(self, backbone_path):
    105.         """ Initialize weights for training. """
    106.         # Initialize the backbone with the pretrained weights.
    107.         self.backbone.init_backbone(backbone_path)
    108.  
    109.         conv_constants = getattr(nn.Conv2d(1, 1, 1), '__constants__')
    110.         
    111.         # Quick lambda to test if one list contains the other
    112.         def all_in(x, y):
    113.             for _x in x:
    114.                 if _x not in y:
    115.                     return False
    116.             return True
    117.  
    118.         # Initialize the rest of the conv layers with xavier
    119.         for name, module in self.named_modules():
    120.             # See issue #127 for why we need such a complicated condition if the module is a WeakScriptModuleProxy
    121.             # Broke in 1.3 (see issue #175), WeakScriptModuleProxy was turned into just ScriptModule.
    122.             # Broke in 1.4 (see issue #292), where RecursiveScriptModule is the new star of the show.
    123.             # Note that this might break with future pytorch updates, so let me know if it does
    124.             is_script_conv = False
    125.             if 'Script' in type(module).__name__:
    126.                 # 1.4 workaround: now there's an original_name member so just use that
    127.                 if hasattr(module, 'original_name'):
    128.                     is_script_conv = 'Conv' in module.original_name
    129.                 # 1.3 workaround: check if this has the same constants as a conv module
    130.                 else:
    131.                     is_script_conv = (
    132.                         all_in(module.__dict__['_constants_set'], conv_constants)
    133.                         and all_in(conv_constants, module.__dict__['_constants_set']))
    134.             
    135.             is_conv_layer = isinstance(module, nn.Conv2d) or is_script_conv
    136.  
    137.             if is_conv_layer and module not in self.backbone.backbone_modules:
    138.                 nn.init.xavier_uniform_(module.weight.data)
    139.  
    140.                 if module.bias is not None:
    141.                     if cfg.use_focal_loss and 'conf_layer' in name:
    142.                         if not cfg.use_sigmoid_focal_loss:
    143.                             # Initialize the last layer as in the focal loss paper.
    144.                             # Because we use softmax and not sigmoid, I had to derive an alternate expression
    145.                             # on a notecard. Define pi to be the probability of outputting a foreground detection.
    146.                             # Then let z = sum(exp(x)) - exp(x_0). Finally let c be the number of foreground classes.
    147.                             # Chugging through the math, this gives us
    148.                             #   x_0 = log(z * (1 - pi) / pi)    where 0 is the background class
    149.                             #   x_i = log(z / c)                for all i > 0
    150.                             # For simplicity (and because we have a degree of freedom here), set z = 1. Then we have
    151.                             #   x_0 =  log((1 - pi) / pi)       note: don't split up the log for numerical stability
    152.                             #   x_i = -log(c)                   for all i > 0
    153.                             module.bias.data[0]  = np.log((1 - cfg.focal_loss_init_pi) / cfg.focal_loss_init_pi)
    154.                             module.bias.data[1:] = -np.log(module.bias.size(0) - 1)
    155.                         else:
    156.                             module.bias.data[0]  = -np.log(cfg.focal_loss_init_pi / (1 - cfg.focal_loss_init_pi))
    157.                             module.bias.data[1:] = -np.log((1 - cfg.focal_loss_init_pi) / cfg.focal_loss_init_pi)
    158.                     else:
    159.                         module.bias.data.zero_()
    160.     
    161.     def train(self, mode=True):
    162.         super().train(mode)
    163.  
    164.         if cfg.freeze_bn:
    165.             self.freeze_bn()
    166.  
    167.     def freeze_bn(self, enable=False):
    168.         """ Adapted from https://discuss.pytorch.org/t/how-to-train-with-frozen-batchnorm/12106/8 """
    169.         for module in self.modules():
    170.             if isinstance(module, nn.BatchNorm2d):
    171.                 module.train() if enable else module.eval()
    172.  
    173.                 module.weight.requires_grad = enable
    174.                 module.bias.requires_grad = enable
    175.     
    176.     def forward(self, x):
    177.         """ The input should be of size [batch_size, 3, img_h, img_w] """
    178.         _, _, img_h, img_w = x.size()
    179.         cfg._tmp_img_h = img_h
    180.         cfg._tmp_img_w = img_w
    181.         
    182.         with timer.env('backbone'):
    183.             outs = self.backbone(x)
    184.  
    185.         if cfg.fpn is not None:
    186.             with timer.env('fpn'):
    187.                 # Use backbone.selected_layers because we overwrote self.selected_layers
    188.                 outs = [outs[i] for i in cfg.backbone.selected_layers]
    189.                 outs = self.fpn(outs)
    190.  
    191.         proto_out = None
    192.         if cfg.mask_type == mask_type.lincomb and cfg.eval_mask_branch:
    193.             with timer.env('proto'):
    194.                 proto_x = x if self.proto_src is None else outs[self.proto_src]
    195.                 
    196.                 if self.num_grids > 0:
    197.                     grids = self.grid.repeat(proto_x.size(0), 1, 1, 1)
    198.                     proto_x = torch.cat([proto_x, grids], dim=1)
    199.  
    200.                 proto_out = self.proto_net(proto_x)
    201.                 proto_out = cfg.mask_proto_prototype_activation(proto_out)
    202.  
    203.                 if cfg.mask_proto_prototypes_as_features:
    204.                     # Clone here because we don't want to permute this, though idk if contiguous makes this unnecessary
    205.                     proto_downsampled = proto_out.clone()
    206.  
    207.                     if cfg.mask_proto_prototypes_as_features_no_grad:
    208.                         proto_downsampled = proto_out.detach()
    209.                 
    210.                 # Move the features last so the multiplication is easy
    211.                 proto_out = proto_out.permute(0, 2, 3, 1).contiguous()
    212.  
    213.                 if cfg.mask_proto_bias:
    214.                     bias_shape = [x for x in proto_out.size()]
    215.                     bias_shape[-1] = 1
    216.                     proto_out = torch.cat([proto_out, torch.ones(*bias_shape)], -1)
    217.  
    218.  
    219.         with timer.env('pred_heads'):
    220.             pred_outs = { 'loc': [], 'conf': [], 'mask': [], 'priors': [] }
    221.  
    222.             if cfg.use_mask_scoring:
    223.                 pred_outs['score'] = []
    224.  
    225.             if cfg.use_instance_coeff:
    226.                 pred_outs['inst'] = []
    227.             
    228.             for idx, pred_layer in zip(self.selected_layers, self.prediction_layers):
    229.                 pred_x = outs[idx]
    230.  
    231.                 if cfg.mask_type == mask_type.lincomb and cfg.mask_proto_prototypes_as_features:
    232.                     # Scale the prototypes down to the current prediction layer's size and add it as inputs
    233.                     proto_downsampled = F.interpolate(proto_downsampled, size=outs[idx].size()[2:], mode='bilinear', align_corners=False)
    234.                     pred_x = torch.cat([pred_x, proto_downsampled], dim=1)
    235.  
    236.                 # A hack for the way dataparallel works
    237.                 if cfg.share_prediction_module and pred_layer is not self.prediction_layers[0]:
    238.                     pred_layer.parent = [self.prediction_layers[0]]
    239.  
    240.                 p = pred_layer(pred_x)
    241.                 
    242.                 for k, v in p.items():
    243.                     pred_outs[k].append(v)
    244.  
    245.         for k, v in pred_outs.items():
    246.             pred_outs[k] = torch.cat(v, -2)
    247.  
    248.         if proto_out is not None:
    249.             pred_outs['proto'] = proto_out
    250.  
    251.         if self.training:
    252.             # For the extra loss functions
    253.             if cfg.use_class_existence_loss:
    254.                 pred_outs['classes'] = self.class_existence_fc(outs[-1].mean(dim=(2, 3)))
    255.  
    256.             if cfg.use_semantic_segmentation_loss:
    257.                 pred_outs['segm'] = self.semantic_seg_conv(outs[0])
    258.  
    259.             return pred_outs
    260.         else:
    261.             if cfg.use_mask_scoring:
    262.                 pred_outs['score'] = torch.sigmoid(pred_outs['score'])
    263.  
    264.             if cfg.use_focal_loss:
    265.                 if cfg.use_sigmoid_focal_loss:
    266.                     # Note: even though conf[0] exists, this mode doesn't train it so don't use it
    267.                     pred_outs['conf'] = torch.sigmoid(pred_outs['conf'])
    268.                     if cfg.use_mask_scoring:
    269.                         pred_outs['conf'] *= pred_outs['score']
    270.                 elif cfg.use_objectness_score:
    271.                     # See focal_loss_sigmoid in multibox_loss.py for details
    272.                     objectness = torch.sigmoid(pred_outs['conf'][:, :, 0])
    273.                     pred_outs['conf'][:, :, 1:] = objectness[:, :, None] * F.softmax(pred_outs['conf'][:, :, 1:], -1)
    274.                     pred_outs['conf'][:, :, 0 ] = 1 - objectness
    275.                 else:
    276.                     pred_outs['conf'] = F.softmax(pred_outs['conf'], -1)
    277.             else:
    278.  
    279.                 if cfg.use_objectness_score:
    280.                     objectness = torch.sigmoid(pred_outs['conf'][:, :, 0])
    281.                     
    282.                     pred_outs['conf'][:, :, 1:] = (objectness > 0.10)[..., None] \
    283.                         * F.softmax(pred_outs['conf'][:, :, 1:], dim=-1)
    284.                     
    285.                 else:
    286.                     pred_outs['conf'] = F.softmax(pred_outs['conf'], -1)
    287.  
    288.             return self.detect(pred_outs, self)

    整理不易，欢迎一键三连！！！


    送你们一条美丽的--分割线--

    🌷🌷🍀🍀🌾🌾🍓🍓🍂🍂🙋🙋🐸🐸🙋🙋💖💖🍌🍌🔔🔔🍉🍉🍭🍭🍋🍋🍇🍇🏆🏆📸📸⛵⛵⭐⭐🍎🍎👍👍🌷🌷

     

  • 相关阅读:
    SQL必需掌握的100个重要知识点：使用子查询
    C++ 练气期之二维数组与矩阵运算
    C语言初学习——易错点合集（持续更新中）
    StarRocks 的学习笔记
    春招秋招，什么是群面和无领导小组讨论
    Jenkins快速了解
    算法打卡01——求两数之和
    YOLOv8改进 | DAttention (DAT)注意力机制实现极限涨点
    目标检测重要概念——IOU、感受野、空洞卷积、mAP
    房地产渠道风控怎么开展
  • 原文地址：https://blog.csdn.net/qq_38308388/article/details/132896153