【MindSpore1.5.0】利用MindSpore1.5.0构建STN模型时出现cudaMalloc failed错误

问题描述：

【功能模块】

[mindspore/ccsrc/runtime/device/gpu/cuda_driver.cc:40] AllocDeviceMem]

[mindspore/ccsrc/runtime/device/gpu/gpu_memory_allocator.cc:89] AllocDeviceMem]

【开发环境】

nvidia-docker容器里运行 nvidia/cuda:10.1-cudnn7-devel-ubuntu18.04镜像，宿主机操作系统是 ubuntu16.04，显卡是TITAN Xp

【操作步骤&问题现象】

1、MindSpore1.5.0-gpu构建STN模型时出现 [mindspore/ccsrc/runtime/device/gpu/cuda_driver.cc:40] AllocDeviceMem] cudaMalloc failed, ret[2], out of memory

以及 [mindspore/ccsrc/runtime/device/gpu/gpu_memory_allocator.cc:89] AllocDeviceMem] Alloc device memory[16711680] failed.

2、

【截图信息】

H、W均为128

 

【日志信息】（可选，上传日志内容或者附件）

[WARNING] ME(12598:140049487251264,MainProcess):2021-12-07-16:49:43.227.879 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(12598:140049487251264,MainProcess):2021-12-07-16:49:43.399.906 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(12598:140049487251264,MainProcess):2021-12-07-16:49:44.256.334 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(12598:140049487251264,MainProcess):2021-12-07-16:49:44.842.445 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[WARNING] ME(12598:140049487251264,MainProcess):2021-12-07-16:49:45.388.572 [mindspore/common/_decorator.py:33] 'TensorAdd' is deprecated from version 1.1 and will be removed in a future version, use 'Add' instead.
[ERROR] DEVICE(12598,7f5fcff04740,python3):2021-12-07-16:50:02.652.640 [mindspore/ccsrc/runtime/device/gpu/cuda_driver.cc:40] AllocDeviceMem] cudaMalloc failed, ret[2], out of memory
[EXCEPTION] DEVICE(12598,7f5fcff04740,python3):2021-12-07-16:50:02.652.703 [mindspore/ccsrc/runtime/device/gpu/gpu_memory_allocator.cc:89] AllocDeviceMem] Alloc device memory[16711680] failed.
Traceback (most recent call last):
  File "./train.py", line 77, in <module>
    net = resnet50(class_num=config.class_num, is_train=True)
  File "/workspace/Masked_Face_Recognition/model/model.py", line 421, in resnet50
    class_num, is_train)
  File "/workspace/Masked_Face_Recognition/model/model.py", line 265, in __init__
    self.STN = STN(128, 128)
  File "/workspace/Masked_Face_Recognition/model/stn.py", line 19, in __init__
    x_t = expand_dims(x_t, 0)
  File "/opt/conda/envs/mindspore1.5.0/lib/python3.7/site-packages/mindspore/ops/primitive.py", line 247, in __call__
    return _run_op(self, self.name, args)
  File "/opt/conda/envs/mindspore1.5.0/lib/python3.7/site-packages/mindspore/common/api.py", line 78, in wrapper
    results = fn(*arg, **kwargs)
  File "/opt/conda/envs/mindspore1.5.0/lib/python3.7/site-packages/mindspore/ops/primitive.py", line 682, in _run_op
    output = real_run_op(obj, op_name, args)
RuntimeError: mindspore/ccsrc/runtime/device/gpu/gpu_memory_allocator.cc:89 AllocDeviceMem] Alloc device memory[16711680] failed.

解答：

看提示就是单纯的内存不够大。。。。

可以考虑换小一些的batch等，部分代码的默认设置，是按照32G显存的设备来设置的，一些其他规格的设备可能需要一些对应的参数调整

可以参考faq中的第一条描述。

models: Models of MindSpore