• 3D视觉——2.人体姿态估计(Pose Estimation)入门——OpenPose含安装、编译、使用（单帧、实时视频）

    本章博客就是对OpenPose工具包进行开发；我呕心沥血（笑哭），经历重重困难，想放弃了很多次（因为openpose的编译实在是太麻烦了）但是后来还是成功了，各位点个赞吧！这个真的太麻烦了。

    按照单帧图像和实时视频的顺序述写，其中单帧是使用的Pytorch编程只是调用OpenPose的模型；实时视频中使用Python调用OpenPose的包，所以必须得安装OpenPose，并对其进行编译，最后再使用。

    首先从github上，下载CMU提供的源码下来：

    https://github.com/CMU-Perceptual-Computing-Lab/openposehttps://github.com/CMU-Perceptual-Computing-Lab/openpose

    项目结构

    1. OpenPose-Demo-Pytorch-master
    2. |
    3. |----images----|----pose.jpg
    4. |----bin(编译之后，从源码拷贝下来的，单帧不看这个) 
    5. |----x64(编译之后，从源码拷贝下来的，单帧不看这个) 
    6. |----Release(编译之后，从源码拷贝下来的，单帧不看这个)
    7. |----models----|----pose----|----body_25----|----pose_deploy.prototxt
    8. |                           |               |----pose_iter_584000.caffemodel
    9. |                           |----coco----|----pose_deploy_linevec.prototxt
    10. |                           |            |----pose_iter_440000.caffemodel
    11. |----video----|----1.mp4
    12. |----config.py
    13. |----predict.py（单帧）
    14. |----Demo.py(实时视频)

    关键点详解

    关键点25（model\pose\body_25\pose_iter_584000.caffemodel or pose_deploy.prototxt）如下图1. 所示，关键点18（model\pose\coco\pose_iter_440000.caffemodel or pose_deploy_linevec.prototxt）如下图2.所示。

    下载模型，可在CMU的github上下载，上面提供了，就不再提供。

    步骤：

    1. git clone https://github.com/CMU-Perceptual-Computing-Lab/openpose.git
    2. or 
    3. downloads .zip
    4. cd openpose-master/models
    5. bash getModels.sh （Linux）
    6. 双击 getModels.bat (Windows)
    7. 下载 pose_iter_584000.caffemodel
    8.      pose_iter_440000.caffemodel
    9. ...(只用这两个，将其放置在我们项目的models\pose\下)

     图1.

    1. {0, “Nose”},
    2. {1, “Neck”},
    3. {2, “RShoulder”},
    4. {3, “RElbow”},
    5. {4, “RWrist”},
    6. {5, “LShoulder”},
    7. {6, “LElbow”},
    8. {7, “LWrist”},
    9. {8, “MidHip”},
    10. {9, “RHip”},
    11. {10, “RKnee”},
    12. {11, “RAnkle”},
    13. {12, “LHip”},
    14. {13, “LKnee”},
    15. {14, “LAnkle”},
    16. {15, “REye”},
    17. {16, “LEye”},
    18. {17, “REar”},
    19. {18, “LEar”},
    20. {19, “LBigToe”},
    21. {20, “LSmallToe”},
    22. {21, “LHeel”},
    23. {22, “RBigToe”},
    24. {23, “RSmallToe”},
    25. {24, “RHeel”}

     图2.

    1. {"Nose": 0, 
    2. "Neck": 1, 
    3. "RShoulder": 2, 
    4. "RElbow": 3, 
    5. "LShoulder": 5, 
    6. "LElbow": 6, 
    7. "LWrist": 7, 
    8. "RHip": 8, 
    9. "RKnee": 9,
    10. "RAnkle": 10, 
    11. "LHip": 11, 
    12. "LKnee": 12, 
    13. "LAnkle": 13, 
    14. "REye": 14,
    15. "LEye": 15, 
    16. "REar": 16, 
    17. "LEar": 17, 
    18. "Background": 18}

    1.单帧代码

    对于单帧将之前的源码下载下来，并将模型权重拷贝（进入源码的models里面双击getModels.bat下载这些权重）到我们自己的项目，就是将models中.prototxt与.caffemodel拷走；之后我们对模型进行推理，其步骤主要为:

    • 首先，读取模型与推理所需要的图像，在进行推理获取结果
    • 其次，关键点检测，再利用PAFs,找到有些关键点对
    • 最后，将点对组合成正确的人体骨骼图

    配置文件

    config.py

    1. prototxt_25 = "models/pose/body_25/pose_deploy.prototxt"
    2. caffemodel_25 = "models/pose/body_25/pose_iter_584000.caffemodel"
    3.  
    4. point_name_25 = ['None', 'Neck', 'RShoulder',
    5.                  'RElbow', 'RWrist', 'LShoulder',
    6.                  'LElbow', 'LWrist', 'MidHip',
    7.                  'RHip', 'RKnee', 'RAnkle',
    8.                  'LHip', 'LKnee', 'LAnkle',
    9.                  'REye', 'LEye', 'REar',
    10.                  'LEar', 'LBigToe', 'LSmallToe',
    11.                  'LHeel', 'RBigToe', 'RSmallToe',
    12.                  'RHeel']
    13. point_pairs_25 = [[1, 8], [1, 2], [1, 5], [2, 3], [3, 4], [5, 6],
    14.                   [6, 7], [8, 9], [9, 10], [10, 11], [8, 12], [12, 13],
    15.                   [13, 14], [1, 0], [0, 15], [15, 17], [0, 16], [16, 18],
    16.                   [2, 17], [5, 18], [14, 19], [19, 20], [14, 21], [11, 22],
    17.                   [22, 23], [11, 24]]
    18.  
    19. map_idx_25 = [[26, 27], [40, 41], [48, 49], [42, 43], [44, 45], [50, 51],
    20.               [52, 53], [32, 33], [28, 29], [30, 31], [34, 35], [36, 37],
    21.               [38, 39], [56, 57], [58, 59], [62, 63], [60, 61], [64, 65],
    22.               [46, 47], [54, 55], [66, 67], [68, 69], [70, 71], [72, 73],
    23.               [74, 75], [76, 77]]
    24.  
    25. colors_25 = [[255, 0, 0], [255, 85, 0], [255, 170, 0],
    26.              [255, 255, 0], [170, 255, 0], [85, 255, 0],
    27.              [0, 255, 0], [0, 255, 85], [0, 255, 170],
    28.              [0, 255, 255], [0, 170, 255], [0, 85, 255],
    29.              [0, 0, 255], [85, 0, 255], [170, 0, 255],
    30.              [255, 0, 255], [255, 0, 170], [255, 0, 85],
    31.              [255, 170, 85], [255, 170, 170], [255, 170, 255],
    32.              [255, 85, 85], [255, 85, 170], [255, 85, 255],
    33.              [170, 170, 170]]
    34.  
    35. prototxt_18 = "./models/coco/pose_deploy_linevec.prototxt"
    36. caffemodel_18 = "./models/coco/pose_iter_440000.caffemodel"
    37.  
    38. point_names_18 = ['Nose', 'Neck',
    39.                   'R-Sho', 'R-Elb', 'R-Wr',
    40.                   'L-Sho', 'L-Elb', 'L-Wr',
    41.                   'R-Hip', 'R-Knee', 'R-Ank',
    42.                   'L-Hip', 'L-Knee', 'L-Ank',
    43.                   'R-Eye', 'L-Eye', 'R-Ear', 'L-Ear']
    44.  
    45. point_pairs_18 = [[1, 2], [1, 5], [2, 3], [3, 4], [5, 6], [6, 7],
    46.                   [1, 8], [8, 9], [9, 10], [1, 11], [11, 12], [12, 13],
    47.                   [1, 0], [0, 14], [14, 16], [0, 15], [15, 17],
    48.                   [2, 17], [5, 16]]
    49.  
    50. map_idx_18 = [[31, 32], [39, 40], [33, 34], [35, 36], [41, 42], [43, 44],
    51.               [19, 20], [21, 22], [23, 24], [25, 26], [27, 28], [29, 30],
    52.               [47, 48], [49, 50], [53, 54], [51, 52], [55, 56],
    53.               [37, 38], [45, 46]]
    54.  
    55. colors_18 = [[0, 100, 255], [0, 100, 255], [0, 255, 255],
    56.              [0, 100, 255], [0, 255, 255], [0, 100, 255],
    57.              [0, 255, 0], [255, 200, 100], [255, 0, 255],
    58.              [0, 255, 0], [255, 200, 100], [255, 0, 255],
    59.              [0, 0, 255], [255, 0, 0], [200, 200, 0],
    60.              [255, 0, 0], [200, 200, 0], [0, 0, 0]]

    OpenPose

    predict.py(核心)

    1. import cv2
    2. import time
    3. import numpy as np
    4. import matplotlib.pyplot as plt
    5. from config import *
    6.  
    7. class general_mulitpose_model(object):
    8.  
    9.     # 初始化 Pose keypoint_num: 25 or 18
    10.     def __init__(self, keypoint_num):
    11.     
    12.     # 加载openpose模型
    13.     def get_model(self):
    14.     
    15.     # 获取关键点
    16.     def getKeypoints(self, probMap, threshold=0.1):
    17.    
    18.     # 获取有效点对
    19.     def getValidPairs(self, output, detected_keypoints, width, height):
    20.    
    21.     # 连接有效点对，获取完整的人体骨骼图
    22.     def getPersonwiseKeypoints(self, valid_pairs, invalid_pairs, keypoints_list):
    23.  
    24.     # 关键点连接后的可视化
    25.     def vis_pose(self, img_file, personwiseKeypoints, keypoints_list):
    26.  
    27.     # 预测（推理）关键点
    28.     def predict(self, imgfile):

    初始化

    1.  def __init__(self, keypoint_num):
    2.         self.point_names = point_name_25 if keypoint_num == 25 else point_names_18
    3.         self.point_pairs = point_pairs_25 if keypoint_num == 25 else point_pairs_18
    4.         self.map_idx = map_idx_25 if keypoint_num == 25 else map_idx_18
    5.         self.colors = colors_25 if keypoint_num == 25 else colors_18
    6.         self.num_points = 25 if keypoint_num == 25 else 18
    7.  
    8.         self.prototxt = prototxt_25 if keypoint_num == 25 else prototxt_18
    9.         self.caffemodel = caffemodel_25 if keypoint_num == 25 else caffemodel_18
    10.         self.pose_net = self.get_model()

    获取关键点

    1.     def getKeypoints(self, probMap, threshold=0.1):
    2.         mapSmooth = cv2.GaussianBlur(probMap, (3, 3), 0, 0)
    3.         mapMask = np.uint8(mapSmooth > threshold)
    4.         keypoints = []
    5.  
    6.         # find the blobs
    7.         contours, hierarchy = cv2.findContours(mapMask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    8.         for cnt in contours:
    9.             blobMask = np.zeros(mapMask.shape)
    10.             blobMask = cv2.fillConvexPoly(blobMask, cnt, 1)
    11.             maskedProbMap = mapSmooth * blobMask
    12.             _, maxVal, _, maxLoc = cv2.minMaxLoc(maskedProbMap)
    13.             keypoints.append(maxLoc + (probMap[maxLoc[1], maxLoc[0]],))
    14.         return keypoints

    获取有效点对

    1.     def getValidPairs(self, output, detected_keypoints, width, height):
    2.         valid_pairs = []
    3.         invalid_pairs = []
    4.         n_interp_samples = 15
    5.         paf_score_th = 0.1
    6.         conf_th = 0.7
    7.  
    8.         for k in range(len(self.map_idx)):
    9.             # A -> B constitute a limb
    10.             pafA = output[0, self.map_idx[k][0], :, :]
    11.             pafB = output[0, self.map_idx[k][1], :, :]
    12.             pafA = cv2.resize(pafA, (width, height))
    13.             pafB = cv2.resize(pafB, (width, height))
    14.  
    15.             candA = detected_keypoints[self.point_pairs[k][0]]
    16.             candB = detected_keypoints[self.point_pairs[k][1]]
    17.             nA = len(candA)
    18.             nB = len(candB)
    19.             if (nA != 0 and nB != 0):
    20.                 valid_pair = np.zeros((0, 3))
    21.                 for i in range(nA):
    22.                     max_j = -1
    23.                     maxScore = -1
    24.                     found = 0
    25.                     for j in range(nB):
    26.                         # Find d_ij
    27.                         d_ij = np.subtract(candB[j][:2], candA[i][:2])
    28.                         norm = np.linalg.norm(d_ij)
    29.                         if norm:
    30.                             d_ij = d_ij / norm
    31.                         else:
    32.                             continue
    33.                         # Find p(u)
    34.                         interp_coord = list(
    35.                             zip(np.linspace(candA[i][0], candB[j][0], num=n_interp_samples),
    36.                                 np.linspace(candA[i][1], candB[j][1], num=n_interp_samples)))
    37.                         # Find L(p(u))
    38.                         paf_interp = []
    39.                         for k in range(len(interp_coord)):
    40.                             paf_interp.append([pafA[int(round(interp_coord[k][1])), int(round(interp_coord[k][0]))],
    41.                                                pafB[int(round(interp_coord[k][1])), int(round(interp_coord[k][0]))]])
    42.                         # Find E
    43.                         paf_scores = np.dot(paf_interp, d_ij)
    44.                         avg_paf_score = sum(paf_scores) / len(paf_scores)
    45.                         # check if the connection is valid
    46.                         # If the fraction of interpolated vectors aligned with PAF is higher then threshold -> Valid Pair
    47.                         if (len(np.where(paf_scores > paf_score_th)[0]) / n_interp_samples) > conf_th:
    48.                             if avg_paf_score > maxScore:
    49.                                 max_j = j
    50.                                 maxScore = avg_paf_score
    51.                                 found = 1
    52.                     # Append the connection to the list
    53.                     if found:
    54.                         valid_pair = np.append(valid_pair, [[candA[i][3], candB[max_j][3], maxScore]], axis=0)
    55.  
    56.                 # Append the detected connections to the global list
    57.                 valid_pairs.append(valid_pair)
    58.  
    59.             else:  # If no keypoints are detected
    60.                 print("No Connection : k = {}".format(k))
    61.                 invalid_pairs.append(k)
    62.                 valid_pairs.append([])
    63.  
    64.         return valid_pairs, invalid_pairs

    连接有效点对，获取完整的人体骨骼图

    1.     def getPersonwiseKeypoints(self, valid_pairs, invalid_pairs, keypoints_list):
    2.         personwiseKeypoints = -1 * np.ones((0, self.num_points + 1))
    3.         for k in range(len(self.map_idx)):
    4.             if k not in invalid_pairs:
    5.                 partAs = valid_pairs[k][:, 0]
    6.                 partBs = valid_pairs[k][:, 1]
    7.                 indexA, indexB = np.array(self.point_pairs[k])
    8.                 for i in range(len(valid_pairs[k])):
    9.                     found = 0
    10.                     person_idx = -1
    11.                     for j in range(len(personwiseKeypoints)):
    12.                         if personwiseKeypoints[j][indexA] == partAs[i]:
    13.                             person_idx = j
    14.                             found = 1
    15.                             break
    16.                     if found:
    17.                         personwiseKeypoints[person_idx][indexB] = partBs[i]
    18.                         personwiseKeypoints[person_idx][-1] += keypoints_list[partBs[i].astype(int), 2] + \
    19.                                                                valid_pairs[k][i][2]
    20.                     elif not found and k < self.num_points - 1:
    21.                         row = -1 * np.ones(self.num_points + 1)
    22.                         row[indexA] = partAs[i]
    23.                         row[indexB] = partBs[i]
    24.                         row[-1] = sum(keypoints_list[valid_pairs[k][i, :2].astype(int), 2]) + \
    25.                                   valid_pairs[k][i][2]
    26.                         personwiseKeypoints = np.vstack([personwiseKeypoints, row])
    27.         return personwiseKeypoints

    关键点连接后的可视化

    1.     def vis_pose(self, img_file, personwiseKeypoints, keypoints_list):
    2.         img = cv2.imread(img_file)
    3.         for i in range(self.num_points - 1):
    4.             for n in range(len(personwiseKeypoints)):
    5.                 index = personwiseKeypoints[n][np.array(self.point_pairs[i])]
    6.                 if -1 in index:
    7.                     continue
    8.                 B = np.int32(keypoints_list[index.astype(int), 0])
    9.                 A = np.int32(keypoints_list[index.astype(int), 1])
    10.                 cv2.line(img, (B[0], A[0]), (B[1], A[1]), self.colors[i], 3, cv2.LINE_AA)
    11.         plt.figure()
    12.         plt.imshow(img[:, :, ::-1])
    13.         plt.title('Results')
    14.         plt.axis("off")
    15.         plt.show()

    预测（推理）关键点

    import cv2 显示

    因为原始图像尺寸太大了，所以我resize了一下。

    1.     def vis_pose(self, img_file, personwiseKeypoints, keypoints_list):
    2.         img = cv2.imread(img_file)
    3.         for i in range(self.num_points - 1):
    4.             for n in range(len(personwiseKeypoints)):
    5.                 index = personwiseKeypoints[n][np.array(self.point_pairs[i])]
    6.                 if -1 in index:
    7.                     continue
    8.                 B = np.int32(keypoints_list[index.astype(int), 0])
    9.                 A = np.int32(keypoints_list[index.astype(int), 1])
    10.                 cv2.line(img, (B[0], A[0]), (B[1], A[1]), self.colors[i], 3, cv2.LINE_AA)
    11.         img = cv2.resize(img, (480, 640))
    12.         cv2.imshow("Results", img)
    13.         cv2.waitKey(0)
    14.         cv2.destroyAllWindows()

    import matplotlib.pyplot as plt 显示

    1.     def predict(self, imgfile):
    2.         img = cv2.imread(imgfile)
    3.         height, width, _ = img.shape
    4.         net_height = 368
    5.         net_width = int((net_height / height) * width)
    6.         start = time.time()
    7.  
    8.         in_blob = cv2.dnn.blobFromImage(
    9.             img, 1.0 / 255, (net_width, net_height), (0, 0, 0), swapRB=False, crop=False)
    10.         self.pose_net.setInput(in_blob)
    11.         output = self.pose_net.forward()
    12.         print("[INFO]Time Taken in Forward pass: {} ".format(time.time() - start))
    13.         detected_keypoints = []
    14.         keypoints_list = np.zeros((0, 3))
    15.         keypoint_id = 0
    16.         threshold = 0.1
    17.         for part in range(self.num_points):
    18.             probMap = output[0, part, :, :]
    19.             probMap = cv2.resize(probMap, (width, height))
    20.  
    21.             keypoints = self.getKeypoints(probMap, threshold)
    22.             print("Keypoints - {} : {}".format(self.point_names[part], keypoints))
    23.             keypoint_with_id = []
    24.             for i in range(len(keypoints)):
    25.                 keypoint_with_id.append(keypoints[i] + (keypoint_id,))
    26.                 keypoints_list = np.vstack([keypoints_list, keypoints[i]])
    27.                 keypoint_id += 1
    28.             detected_keypoints.append(keypoint_with_id)
    29.         valid_paris, invalid_pairs = self.getValidPairs(output, detected_keypoints, width, height)
    30.         personwiseKeypoints = self.getPersonwiseKeypoints(valid_paris, invalid_pairs, keypoints_list)
    31.         self.vis_pose(imgfile, personwiseKeypoints, keypoints_list)

    main.py

    1. if __name__ == '__main__':
    2.     gmm = general_mulitpose_model(25)
    3.     personwiseKeypoints, keypoints_list = gmm.predict("images/pose.jpg")

    完整代码

    1. import cv2
    2. import time
    3. import math
    4. import numpy as np
    5. from config import *
    6.  
    7.  
    8. class general_mulitpose_model(object):
    9.     def __init__(self, keypoint_num):
    10.         self.point_names = point_name_25 if keypoint_num == 25 else point_names_18
    11.         self.point_pairs = point_pairs_25 if keypoint_num == 25 else point_pairs_18
    12.         self.map_idx = map_idx_25 if keypoint_num == 25 else map_idx_18
    13.         self.colors = colors_25 if keypoint_num == 25 else colors_18
    14.         self.num_points = 25 if keypoint_num == 25 else 18
    15.  
    16.         self.prototxt = prototxt_25 if keypoint_num == 25 else prototxt_18
    17.         self.caffemodel = caffemodel_25 if keypoint_num == 25 else caffemodel_18
    18.         self.pose_net = self.get_model()
    19.  
    20.     def get_model(self):
    21.         coco_net = cv2.dnn.readNetFromCaffe(self.prototxt, self.caffemodel)
    22.         return coco_net
    23.  
    24.     def predict(self, imgfile):
    25.         start = time.time()
    26.         img = cv2.imread(imgfile)
    27.         height, width, _ = img.shape
    28.         net_height = 368
    29.         net_width = int((net_height / height) * width)
    30.         start = time.time()
    31.  
    32.         in_blob = cv2.dnn.blobFromImage(
    33.             img, 1.0 / 255, (net_width, net_height), (0, 0, 0), swapRB=False, crop=False)
    34.         self.pose_net.setInput(in_blob)
    35.         output = self.pose_net.forward()
    36.         print("[INFO]Time Taken in Forward pass: {} ".format(time.time() - start))
    37.         detected_keypoints = []
    38.         keypoints_list = np.zeros((0, 3))
    39.         keypoint_id = 0
    40.         threshold = 0.1
    41.         for part in range(self.num_points):
    42.             probMap = output[0, part, :, :]
    43.             probMap = cv2.resize(probMap, (width, height))
    44.  
    45.             keypoints = self.getKeypoints(probMap, threshold)
    46.             print("Keypoints - {} : {}".format(self.point_names[part], keypoints))
    47.             keypoint_with_id = []
    48.             for i in range(len(keypoints)):
    49.                 keypoint_with_id.append(keypoints[i] + (keypoint_id,))
    50.                 keypoints_list = np.vstack([keypoints_list, keypoints[i]])
    51.                 keypoint_id += 1
    52.             detected_keypoints.append(keypoint_with_id)
    53.         valid_paris, invalid_pairs = self.getValidPairs(output, detected_keypoints, width, height)
    54.         personwiseKeypoints = self.getPersonwiseKeypoints(valid_paris, invalid_pairs, keypoints_list)
    55.         img = self.vis_pose(imgfile, personwiseKeypoints, keypoints_list)
    56.         FPS = math.ceil(1 / (time.time() - start))
    57.         img = cv2.putText(img, "FPS" + str(int(FPS)), (25, 50), cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 3)
    58.         return img
    59.  
    60.     def getKeypoints(self, probMap, threshold=0.1):
    61.         mapSmooth = cv2.GaussianBlur(probMap, (3, 3), 0, 0)
    62.         mapMask = np.uint8(mapSmooth > threshold)
    63.         keypoints = []
    64.  
    65.         # find the blobs
    66.         _, contours, hierarchy = cv2.findContours(mapMask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)
    67.         for cnt in contours:
    68.             blobMask = np.zeros(mapMask.shape)
    69.             blobMask = cv2.fillConvexPoly(blobMask, cnt, 1)
    70.             maskedProbMap = mapSmooth * blobMask
    71.             _, maxVal, _, maxLoc = cv2.minMaxLoc(maskedProbMap)
    72.             keypoints.append(maxLoc + (probMap[maxLoc[1], maxLoc[0]],))
    73.         return keypoints
    74.  
    75.     def getValidPairs(self, output, detected_keypoints, width, height):
    76.         valid_pairs = []
    77.         invalid_pairs = []
    78.         n_interp_samples = 15
    79.         paf_score_th = 0.1
    80.         conf_th = 0.7
    81.  
    82.         for k in range(len(self.map_idx)):
    83.             # A -> B constitute a limb
    84.             pafA = output[0, self.map_idx[k][0], :, :]
    85.             pafB = output[0, self.map_idx[k][1], :, :]
    86.             pafA = cv2.resize(pafA, (width, height))
    87.             pafB = cv2.resize(pafB, (width, height))
    88.  
    89.             candA = detected_keypoints[self.point_pairs[k][0]]
    90.             candB = detected_keypoints[self.point_pairs[k][1]]
    91.             nA = len(candA)
    92.             nB = len(candB)
    93.             if (nA != 0 and nB != 0):
    94.                 valid_pair = np.zeros((0, 3))
    95.                 for i in range(nA):
    96.                     max_j = -1
    97.                     maxScore = -1
    98.                     found = 0
    99.                     for j in range(nB):
    100.                         # Find d_ij
    101.                         d_ij = np.subtract(candB[j][:2], candA[i][:2])
    102.                         norm = np.linalg.norm(d_ij)
    103.                         if norm:
    104.                             d_ij = d_ij / norm
    105.                         else:
    106.                             continue
    107.                         # Find p(u)
    108.                         interp_coord = list(
    109.                             zip(np.linspace(candA[i][0], candB[j][0], num=n_interp_samples),
    110.                                 np.linspace(candA[i][1], candB[j][1], num=n_interp_samples)))
    111.                         # Find L(p(u))
    112.                         paf_interp = []
    113.                         for k in range(len(interp_coord)):
    114.                             paf_interp.append([pafA[int(round(interp_coord[k][1])), int(round(interp_coord[k][0]))],
    115.                                                pafB[int(round(interp_coord[k][1])), int(round(interp_coord[k][0]))]])
    116.                         # Find E
    117.                         paf_scores = np.dot(paf_interp, d_ij)
    118.                         avg_paf_score = sum(paf_scores) / len(paf_scores)
    119.                         # check if the connection is valid
    120.                         # If the fraction of interpolated vectors aligned with PAF is higher then threshold -> Valid Pair
    121.                         if (len(np.where(paf_scores > paf_score_th)[0]) / n_interp_samples) > conf_th:
    122.                             if avg_paf_score > maxScore:
    123.                                 max_j = j
    124.                                 maxScore = avg_paf_score
    125.                                 found = 1
    126.                     # Append the connection to the list
    127.                     if found:
    128.                         valid_pair = np.append(valid_pair, [[candA[i][3], candB[max_j][3], maxScore]], axis=0)
    129.  
    130.                 # Append the detected connections to the global list
    131.                 valid_pairs.append(valid_pair)
    132.  
    133.             else:  # If no keypoints are detected
    134.                 print("No Connection : k = {}".format(k))
    135.                 invalid_pairs.append(k)
    136.                 valid_pairs.append([])
    137.  
    138.         return valid_pairs, invalid_pairs
    139.  
    140.     def getPersonwiseKeypoints(self, valid_pairs, invalid_pairs, keypoints_list):
    141.         personwiseKeypoints = -1 * np.ones((0, self.num_points + 1))
    142.         for k in range(len(self.map_idx)):
    143.             if k not in invalid_pairs:
    144.                 partAs = valid_pairs[k][:, 0]
    145.                 partBs = valid_pairs[k][:, 1]
    146.                 indexA, indexB = np.array(self.point_pairs[k])
    147.                 for i in range(len(valid_pairs[k])):
    148.                     found = 0
    149.                     person_idx = -1
    150.                     for j in range(len(personwiseKeypoints)):
    151.                         if personwiseKeypoints[j][indexA] == partAs[i]:
    152.                             person_idx = j
    153.                             found = 1
    154.                             break
    155.                     if found:
    156.                         personwiseKeypoints[person_idx][indexB] = partBs[i]
    157.                         personwiseKeypoints[person_idx][-1] += keypoints_list[partBs[i].astype(int), 2] + \
    158.                                                                valid_pairs[k][i][2]
    159.                     elif not found and k < self.num_points - 1:
    160.                         row = -1 * np.ones(self.num_points + 1)
    161.                         row[indexA] = partAs[i]
    162.                         row[indexB] = partBs[i]
    163.                         row[-1] = sum(keypoints_list[valid_pairs[k][i, :2].astype(int), 2]) + \
    164.                                   valid_pairs[k][i][2]
    165.                         personwiseKeypoints = np.vstack([personwiseKeypoints, row])
    166.         return personwiseKeypoints
    167.  
    168.     def vis_pose(self, img_file, personwiseKeypoints, keypoints_list):
    169.         img = cv2.imread(img_file)
    170.         for i in range(self.num_points - 1):
    171.             for n in range(len(personwiseKeypoints)):
    172.                 index = personwiseKeypoints[n][np.array(self.point_pairs[i])]
    173.                 if -1 in index:
    174.                     continue
    175.                 B = np.int32(keypoints_list[index.astype(int), 0])
    176.                 A = np.int32(keypoints_list[index.astype(int), 1])
    177.                 cv2.line(img, (B[0], A[0]), (B[1], A[1]), self.colors[i], 3, cv2.LINE_AA)
    178.         img = cv2.resize(img, (480, 640))
    179.         return img
    180.  
    181.  
    182. if __name__ == '__main__':
    183.     gmm = general_mulitpose_model(25)
    184.     img = gmm.predict("images/pose.jpg")
    185.     cv2.imshow("frame", img)
    186.     cv2.waitKey(0)
    187.     cv2.destroyAllWindows()

    运行结果

    cv2显示

    plt 显示

    2.实时视频

    因为之前都只是调用了openpose的模型并没有真正使用源码，所以现在真正使用，并且编译一下，其步骤为：

    1）配置文件3rdparty\windows

    将之前github上下载好的项目，找到位置打开，如我的位置：

    D:\PycharmProject\openpose-master

    进入"3rdparty"，找到windows，双击四个.bat文件

    1. D:\PycharmProject\openpose-master\3rdparty\windows
    2. getCaffe.bat
    3. getCaffe3rdparty.bat
    4. getFreeglut.bat
    5. getOpenCV.bat

     

     2）配置文件3rdparty\caffe or pybind11

    进入官网的"3rdparty"，找到caffe or pybind11

     将其git clone https://github.com/CMU-Perceptual-Computing-Lab/caffe.git 或者 下载.zip文件， 放到你文件所在的位置如：

    'D:\PycharmProject\openpose-master\3rdparty\caffe'

    将其git clone https://github.com/pybind/pybind11.git 或者 下载.zip文件，放到你文件所在的位置如：'D:\PycharmProject\openpose-master\3rdparty\pybind11'

    如图

    3）模型下载（之前已经介绍过了）

    1. cd openpose-master/models
    2. bash getModels.sh （Linux）
    3. 双击 getModels.bat (Windows)
    4. 下载 pose_iter_584000.caffemodel
    5.      pose_iter_440000.caffemodel
    6. ...(还有hand，face的模型)

    4）Cmake编译

    首先下载cmake-gui:

    https://cmake.org/download/https://cmake.org/download/windows就下载.msi版本的

    之后就是将openpose-master编译

    第三行的build是自己取的名字，可以直接build或者其他build_CPU

     点击Add Entry，输入自己的Python路径，再点击OK!

     之后，点击“Configure“

    配置vs，你的vs要和你电脑的版本一样，可在 控制面板-> 程序 中查看

     完成之后，再点BUILD_PYTHON,DOWNLOAD_BODY_25_MODEL，DOWNLOAD_BODY_COCO_MODEL,DOWNLOAD_BODY_MPI_MODEL（hand,face也如果有用也选吧！)。

     

    “GPU_MODE”选中“CPU_ONLY”,不选"USE_CUDNN"；你也可以选择"CUDA",那之后必须选择“USE_CUDNN”

     点击“Configure”，等全部完成之后，点击“Generate”

    5）编译工程

    找到openpose-master/build/OpenPose.sln使用vs 2017打开，输入(release x64版本)点击绿色倒三角符号，等待结果

     如果成功这是下面这种状态，并且视频摄像头打开，openpose开始识别人体姿态与人！

     之后右键点击pyopenpose，设为启动项目

     之后结合，官网给的代码，仿照"openpose-master\build\examples\tutorial_api_python\01_body_from_image.py"来导入pyopenpose

    把官网给的openpose-master\build\bin 与 openpose-master\x64拷贝到自己的项目里面去

    把openpose-master\build\python\openpose\Release 导入自己的项目

    再把openpose-master\models中的 hand 和 face 还有 pose 导入自己的项目中去 

    代码

    尝试导入openpose，查看是否成功

    1. import os
    2. import sys
    3. from sys import platform
    4.  
    5. BASE_DIR = os.path.dirname(os.path.realpath(__file__))
    6. if platform == 'win32':
    7.     lib_dir = 'Release'
    8.     bin_dir = 'bin'
    9.     x64_dir = 'x64'
    10.     lib_path = os.path.join(BASE_DIR, lib_dir)
    11.     bin_path = os.path.join(BASE_DIR, bin_dir)
    12.     x64_path = os.path.join(BASE_DIR, x64_dir)
    13.     sys.path.append(lib_path)
    14.     os.environ['PATH'] += ';' + bin_path + ';' + x64_path + '\Release;'
    15.     try:
    16.         import pyopenpose as op
    17.         print("successful, import pyopenpose!")
    18.     except ImportError as e:
    19.         print("fail to import pyopenpose!")
    20.         raise e
    21. else:
    22.     print(f"当前电脑环境:\n{platform}\n")
    23.     sys.exit(-1)

     查看结果

    实时视频核心代码

    1.     # 处理数据
    2.     datum = op.Datum()
    3.  
    4.     # 开始openpose
    5.     opWrapper = op.WrapperPython()
    6.    
    7.     # 配置参数
    8.     params = dict()
    9.     params["model_folder"] = BASE_DIR + "\models"
    10.     params["model_pose"] = "BODY_25"
    11.     params["number_people_max"] = 3
    12.     params["disable_blending"] = False
    13.  
    14.     # 导入参数
    15.     opWrapper.configure(params)
    16.     opWrapper.start()
    17. ...

    1. ...                
    2.                 # 处理图像
    3.  
    4.                 # 输入图像frame打入datum.cvInputData
    5.                 datum.cvInputData = frame
    6.                 # 处理输入图像
    7.                 opWrapper.emplaceAndPop(op.VectorDatum([datum]))
    8.                 # 输出图像为opframe
    9.                 opframe = datum.cvOutputData
    10. ....

     完整代码

    1. import os
    2. import time
    3.  
    4. import cv2
    5. import sys
    6. from tqdm import tqdm
    7. from sys import platform
    8.  
    9. BASE_DIR = os.path.dirname(os.path.realpath(__file__))
    10. if platform == 'win32':
    11.     lib_dir = 'Release'
    12.     bin_dir = 'bin'
    13.     x64_dir = 'x64'
    14.     lib_path = os.path.join(BASE_DIR, lib_dir)
    15.     bin_path = os.path.join(BASE_DIR, bin_dir)
    16.     x64_path = os.path.join(BASE_DIR, x64_dir)
    17.     sys.path.append(lib_path)
    18.     os.environ['PATH'] += ';' + bin_path + ';' + x64_path + '\Release;'
    19.     try:
    20.         import pyopenpose as op
    21.  
    22.         print("successful, import pyopenpose!")
    23.     except ImportError as e:
    24.         print("fail to import pyopenpose!")
    25.         raise e
    26. else:
    27.     print(f"当前电脑环境:\n{platform}\n")
    28.     sys.exit(-1)
    29.  
    30.  
    31. def out_video(input):
    32.     datum = op.Datum()
    33.     opWrapper = op.WrapperPython()
    34.     params = dict()
    35.     params["model_folder"] = BASE_DIR + "\models"
    36.     params["model_pose"] = "BODY_25"
    37.     params["number_people_max"] = 3
    38.     params["disable_blending"] = False
    39.     opWrapper.configure(params)
    40.     opWrapper.start()
    41.     file = input.split("/")[-1]
    42.     output = "video/out-optim-" + file
    43.     print("It will start processing video: {}".format(input))
    44.     cap = cv2.VideoCapture(input)
    45.     frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    46.     frame_size = (int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)), int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT)))
    47.     # create VideoWriter,VideoWriter_fourcc is video decode
    48.     fourcc = cv2.VideoWriter_fourcc('D', 'I', 'V', 'X')
    49.     fps = cap.get(cv2.CAP_PROP_FPS)
    50.     out = cv2.VideoWriter(output, fourcc, fps, frame_size)
    51.     # the progress bar
    52.     with tqdm(range(frame_count)) as pbar:
    53.  
    54.         while cap.isOpened():
    55.             start = time.time()
    56.             success, frame = cap.read()
    57.             if success:
    58.                 datum.cvInputData = frame
    59.                 opWrapper.emplaceAndPop(op.VectorDatum([datum]))
    60.                 opframe = datum.cvOutputData
    61.                 FPS = 1 / (time.time() - start)
    62.                 opframe = cv2.putText(opframe, "FPS" + str(int(FPS)), (25, 50), cv2.FONT_HERSHEY_SIMPLEX, 1,
    63.                                       (0, 255, 0), 3)
    64.                 out.write(opframe)
    65.                 pbar.update(1)
    66.             else:
    67.                 break
    68.  
    69.     pbar.close()
    70.     cv2.destroyAllWindows()
    71.     out.release()
    72.     cap.release()
    73.     print("{} finished!".format(output))
    74.  
    75.  
    76. if __name__ == "__main__":
    77.     video_dir = "video/2.avi"
    78.     out_video(video_dir)

    运行结果

    OpenPose运行结果

    效果比之前的MediaPipe好很多

    参考：

    工程实现 || 基于opencv使用openpose完成人体姿态估计https://blog.csdn.net/magic_ll/article/details/108451560?spm=1001.2014.3001.5506openpose从安装到实战全攻略！(win10)https://zhuanlan.zhihu.com/p/500651669

  • 相关阅读:
    二叉搜索树的实现
    Data-free NAS
    ES6中Promise用法（封装Ajax）
    节假日和汉语拼音接口
    通过uboot命令开启核隔离
    AJAX概念
    【ASP.NET Core】标记帮助器——替换元素名称
    【毕业设计】基于javaEE+SSH+SQL Server的学生信息管理系统设计与实现（毕业论文+程序源码）——学生信息管理系统
    给女生说一些关于软件测试职场的一些真实的建议~
    金蝶云星空企业版v8.0内网穿透配置详解：实现便捷的异地远程访问
  • 原文地址：https://blog.csdn.net/XiaoyYidiaodiao/article/details/125565738