Dlib 是一个 C++ 工具包,被广泛应用于工业和学术界。Dlib 的开源许可允许在任何应用程序中免费使用它。Dlib支持导出其他编程语言如Python的binding。
在Python环境下一般安装dlib很方便,直接使用pip安装即可。但是某些时候由于CUDA支持的问题导致部分模型使用dlib库进行推理会出现异常。
本文简要介绍无CUDA支持的dlib库的安装与使用。
以Ubuntu为例进行说明:
pip install dlib
注:使用此方式安装的dlib默认支持CUDA加速(如已安装CUDA和CUDNN)。
>>> import dlib
>>> dlib.DLIB_USE_CUDA
True
但是在使用face_recognition库进行推理的时候,出现了多卡机器上的显存分配异常(如下图中的253MiB的异常显存占用)和illegal memory access was encountered
等问题。
| 3 N/A N/A 46077 C ...envs/test/bin/python 4409MiB |
| 3 N/A N/A 46115 C ...envs/test/bin/python 8569MiB |
| 3 N/A N/A 46165 C ...envs/test/bin/python 5822MiB |
| 3 N/A N/A 46176 C ...envs/test/bin/python 253MiB |
| 3 N/A N/A 46186 C ...envs/test/bin/python 253MiB |
| 3 N/A N/A 46192 C ...envs/test/bin/python 253MiB |
| 4 N/A N/A 46089 C ...envs/test/bin/python 4409MiB |
| 4 N/A N/A 46118 C ...envs/test/bin/python 8569MiB |
| 4 N/A N/A 46176 C ...envs/test/bin/python 5569MiB |
| 6 N/A N/A 46100 C ...envs/test/bin/python 4409MiB |
| 6 N/A N/A 46125 C ...envs/test/bin/python 8569MiB |
| 6 N/A N/A 46186 C ...envs/test/bin/python 5569MiB |
| 7 N/A N/A 46111 C ...envs/test/bin/python 4409MiB |
| 7 N/A N/A 46150 C ...envs/test/bin/python 8569MiB |
while calling cudnnFindConvolutionForwardAlgorithm( context(),
descriptor(data), (const cudnnFilterDescriptor_t)filter_handle,
(const cudnnConvolutionDescriptor_t)conv_handle,
descriptor(dest_desc), num_possible_algorithms, &num_algorithms,
perf_results.data()) in file /tmp/pip-install-fz7s/dlib_537e10d/dlib/cuda/cudnn_dlibapi.cpp:819.
code: 2, reason: CUDA Resources could not be allocated.
6846:CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
6847:For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
6889:2022-11-04 18:08:52,712 CUDA error: an illegal memory access was encountered
6890:CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
6891:For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
进一步分析发现,是由于机器的GPU和显卡驱动以及CUDNN太新,而face_recognition中的部分模型使用dlib导入后,会引发上述异常。
同样在另一台旧机器上则不会出现上述问题,分析发现是同样版本的dlib在那台机器上没有支持CUDA,于是也就不会出现上述显存异常问题。
>>> import dlib
>>> dlib.DLIB_USE_CUDA
False
于是可以考虑编译安装不支持CUDA的dlib以正常使用face_recognition。
git clone https://github.com/davisking/dlib.git
cd dlib
python setup.py install --no DLIB_USE_CUDA
测试:
>>> import dlib
>>> dlib.DLIB_USE_CUDA
False
import numpy as np
import face_recognition
img = np.zeros((100,100,3)).astype(np.uint8)
face_encodings = face_recognition.face_encodings(img, known_face_locations=[[10, 50, 50, 10]], model="small")
运行正常。
face_encodings = face_recognition.face_encodings(img[:, : , ::-1], known_face_locations=[[10, 50, 50, 10]], model="small")
运行会报错:
Traceback (most recent call last):
File "" , line 1, in <module>
File "/data1/miniconda3/envs/torch1.12/lib/python3.7/site-packages/face_recognition/api.py", line 214, in face_encodings
return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landmark_set, num_jitters)) for raw_landmark_set in raw_landmarks]
File "/data1/miniconda3/envs/torch1.12/lib/python3.7/site-packages/face_recognition/api.py", line 214, in <listcomp>
return [np.array(face_encoder.compute_face_descriptor(face_image, raw_landmark_set, num_jitters)) for raw_landmark_set in raw_landmarks]
TypeError: compute_face_descriptor(): incompatible function arguments. The following argument types are supported:
1. (self: _dlib_pybind11.face_recognition_model_v1, img: numpy.ndarray[(rows,cols,3),numpy.uint8], face: _dlib_pybind11.full_object_detection, num_jitters: int = 0, padding: float = 0.25) -> _dlib_pybind11.vector
2. (self: _dlib_pybind11.face_recognition_model_v1, img: numpy.ndarray[(rows,cols,3),numpy.uint8], num_jitters: int = 0) -> _dlib_pybind11.vector
3. (self: _dlib_pybind11.face_recognition_model_v1, img: numpy.ndarray[(rows,cols,3),numpy.uint8], faces: _dlib_pybind11.full_object_detections, num_jitters: int = 0, padding: float = 0.25) -> _dlib_pybind11.vectors
4. (self: _dlib_pybind11.face_recognition_model_v1, batch_img: List[numpy.ndarray[(rows,cols,3),numpy.uint8]], batch_faces: List[_dlib_pybind11.full_object_detections], num_jitters: int = 0, padding: float = 0.25) -> _dlib_pybind11.vectorss
5. (self: _dlib_pybind11.face_recognition_model_v1, batch_img: List[numpy.ndarray[(rows,cols,3),numpy.uint8]], num_jitters: int = 0) -> _dlib_pybind11.vectors
Invoked with: <_dlib_pybind11.face_recognition_model_v1 object at 0x7f746912a370>, array([[[0, 0, 0],
[0, 0, 0],
[0, 0, 0],
img = np.ascontiguousarray(img[:, :, ::-1])
face_encodings = face_recognition.face_encodings(img, known_face_locations=[[10, 50, 50, 10]], model="small")
运行正常。
如果在最新机器上使用基于dlib的face_recognition库推理,同时使用tensorflow进行其他模型的推理,则tensorflow可能也会有CUDA相关异常的打印,但是溯源还是由于dlib的CUDA支持不正常引发的问题:
2022-11-04 15:32:20.964078: E tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:723] failed to record completion event; therefore, failed to create inter-stream dependency
2022-11-04 15:32:20.964146: E tensorflow/stream_executor/cuda/cuda_driver.cc:1183] failed to enqueue async memcpy from host to device: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; GPU dst: 0x7f6b56e84a00; host src: 0x55bf1eaa4cc0; size: 602112=0x93000
2022-11-04 15:32:20.964171: E tensorflow/stream_executor/stream.cc:334] Error recording event in stream: Error recording CUDA event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered; not marking stream as bad, as the Event object may be at fault. Monitor for further errors.
2022-11-04 15:32:20.964341: E tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2022-11-04 15:32:20.964352: F tensorflow/core/common_runtime/device/device_event_mgr.cc:221] Unexpected Event status: 1
本文为原创文章,独家发布在blog.csdn.net/TracelessLe。未经个人允许不得转载。如需帮助请email至tracelessle@163.com或扫描个人介绍栏二维码咨询。
[1] dlib C++ Library - Frequently Asked Questions
[2] davisking/dlib: A toolkit for making real world machine learning and data analysis applications in C++
[3] How to force install Dlib with only CPU support on a GPU machine with Cuda enabled · Issue #1885 · davisking/dlib
[4] ageitgey/face_recognition: The world’s simplest facial recognition api for Python and the command line