MindSpore:【模型训练】gpu在训练结束后运行model.时，若打开了混合精度就会报数据类型不兼容

问题描述：

【功能模块】

mindspore

【操作步骤&问题现象】

1、用cifar10数据集训练网络

2、model.eval(dataset_eval)

在NPU上正常。

在GPU上：如果amp_level=O0，则正常返回准确率；如果amp_level=O2，就会打印如下报错信息：

[ERROR] DEVICE(28009,python3.7):2021-03-25-17:14:07.696.542 [mindspore/ccsrc/runtime/device/gpu/kernel_info_setter.cc:118] SelectAkgKernel] Not find op[BatchNorm] in akg

[ERROR] DEVICE(28009,python3.7):2021-03-25-17:14:07.696.594 [mindspore/ccsrc/runtime/device/gpu/kernel_info_setter.cc:322] PrintUnsupportedTypeException] Select GPU kernel op[BatchNorm] fail! Incom

patible data type!

The supported data types are in[float32 float32 float32 float32 float32], out[float32 float32 float32 float32 float32]; in[float16 float32 float32 float32 float32], out[float16 float32 float32 floa

t32 float32]; , but get in [float16 float16 float16 float16 float16 ] out [float16 float16 float16 float16 float16 ]

Traceback (most recent call last):

  File "train.py", line 150, in

    acc = model.eval(ds_eval)

  File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/train/model.py", line 717, in eval

    return self._eval_dataset_sink_process(valid_dataset, list_callback, cb_params)

  File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/train/model.py", line 621, in _eval_dataset_sink_process

    outputs = self._eval_network(*inputs)

  File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 322, in __call__

    out = self.compile_and_run(*inputs)

  File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 578, in compile_and_run

    self.compile(*inputs)

  File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/nn/cell.py", line 565, in compile

    _executor.compile(self, *inputs, phase=self.phase, auto_parallel_mode=self._auto_parallel_mode)

  File "/opt/anaconda3/envs/python3.7/lib/python3.7/site-packages/mindspore/common/api.py", line 505, in compile

    result = self._executor.compile(obj, args_list, phase, use_vm)

TypeError: mindspore/ccsrc/runtime/device/gpu/kernel_info_setter.cc:322 PrintUnsupportedTypeException] Select GPU kernel op[BatchNorm] fail! Incompatible data type!

The supported data types are in[float32 float32 float32 float32 float32], out[float32 float32 float32 float32 float32]; in[float16 float32 float32 float32 float32], out[float16 float32 float32 floa

t32 float32]; , but get in [float16 float16 float16 float16 float16 ] out [float16 float16 float16 float16 float16 ]

# In file /home/huawei/longtingting/ms-seng/seng_layer.py(593)

            output = self.matmul_bt(x, self.weight)

【截图信息】

amp_level=O2时的报错信息：

【日志信息】（可选，上传日志内容或者附件）

 

解决方案：

GPU推理中的BatchNorm算子不支持纯fp16输入，推理应该不需要开启混合精度。 所以你推理时的model不要共用训练时候的。

一般有两种方式去得到精度结果：1 使用独立的eval过程，重新定义eval的net并对保存的ckpt进行推理。2 边训练边推理，这种方式需要使用回调函数进行验证。 这两种方式都可以参考master上，model_zoo/offical/cv/resnet/中的用法。