ubuntu 20.04 Kimera semantic 运行记录

Ubuntu20.04 Kimera Semantic运行记录

Kimera VIO ROS

配置

MIT Kimera-VIO-ROS 安装

mkdir -p Kimera_ws/src
cd Kimera_ws
catkin init
catkin config --cmake-args -DCMAKE_BUILD_TYPE=Release -DGTSAM_TANGENT_PREINTEGRATION=OFF
catkin config --merge-devel

cd src
git clone https://github.com/ToniRV/mesh_rviz_plugins.git	
git clone https://github.com/MIT-SPARK/Kimera-VIO-ROS.git

# 切换版本
cd Kimera-VIO-ROS
git checkout 9b8aeae

cd ..
wstool init
wstool merge Kimera-VIO-ROS/install/kimera_vio_ros_https.rosinstall
wstool update

# 更换gtsam版本
rm -rf gtsam
# zph环境
git clone -b 4.1.1 https://github.com/borglab/gtsam.git

# 更换Kimera-RPGO版本
cd Kimera-RPGO
git checkout 9e1bf93

# 更换Kimera-VIO版本
cd ../Kimera-VIO
git checkout 641576f

# 更换Kimera-VIO-ROS版本
cd ../Kimera-VIO-ROS
git checkout 9b8aeae

# 编译
catkin build
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

测试

roslaunch kimera_vio_ros kimera_vio_ros_euroc.launch online:=true run_stereo_dense:=true	
rviz -d $(rospack find kimera_vio_ros)/rviz/kimera_vio_euroc.rviz
rosbag play --clock MH_01_easy.bag
1
2
3

d455 运行

1. d455_camchain.yaml
   复制一份Kimera_VIO_catkin_ws/src/Kimera-VIO-ROS/cfg/calib/euroc_camchain.yaml改为d455_camchain.yaml，并将相关参数填入
2. kimera_vio_ros_d455.launch
   复制一份Kimera_VIO_catkin_ws/src/Kimera-VIO-ROS/launch/kimera_vio_ros_euroc.launch改为kimera_vio_ros_d455.launch，并修改：

1

# 如果是rosbag需要注意改动

 
1
2
3

1
2

      

	  
      
      
      
1
2
3
4
5
6

1
2

添加，参数按需修改

	

   
  
  
  
  
  
1
2
3
4
5
6
7
8

3. kimera_vio_d455.rviz
   复制一份Kimera_VIO_catkin_ws/src/Kimera-VIO-ROS/rviz/kimera_vio_euroc.rviz改为kimera_vio_d455.rviz
   可以在后续打开后修改保存

4. D455相机参数
   复制一份Kimera_VIO_catkin_ws/src/Kimera-VIO/params/Euroc改为D455，在对应的LeftCameraParams.yaml、RightCameraParams.yaml、ImuParams.yaml填入相机参数

运行结果

roslaunch kimera_vio_ros kimera_vio_ros_d455.launch run_stereo_dense:=true	
rviz -d $(rospack find kimera_vio_ros)/rviz/kimera_vio_d455.rviz
1
2

Kimera semantic

以下基本为官方教程

sudo apt-get install python3-wstool python3-catkin-tools  protobuf-compiler autoconf
sudo apt-get install ros-noetic-cmake-modules
1
2

# Setup catkin workspace
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws/
catkin init
catkin config --extend /opt/ros/noetic
catkin config --cmake-args -DCMAKE_BUILD_TYPE=Release
catkin config --merge-devel

# Add workspace to bashrc.
echo 'source ~/catkin_ws/devel/setup.bash' >> ~/.bashrc

# Clone repo
cd ~/catkin_ws/src
git clone https://github.com/MIT-SPARK/Kimera-Semantics.git

# 添加mesh
git clone https://github.com/ToniRV/mesh_rviz_plugins.git	

# Install dependencies from rosinstall file using wstool
wstool init # Use unless wstool is already initialized

# Optionally add Kimera-Semantics to the rosinstall file
# wstool scrape

# For https: 
wstool merge Kimera-Semantics/install/kimera_semantics_https.rosinstall

# Download and update all dependencies
wstool update
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

# Compile code
catkin build kimera_semantics_ros mesh_rviz_plugins # 见ERROR1

# Refresh workspace
source ~/catkin_ws/devel/setup.bash
1
2
3
4
5

测试

roslaunch kimera_vio_ros kimera_vio_ros_euroc.launch run_stereo_dense:=true
roslaunch kimera_semantics_ros kimera_semantics_euroc.launch
rviz -d $(rospack find kimera_semantics_ros)/rviz/kimera_semantics_euroc.rviz
rosbag play --clock MH_01_easy.bag
1
2
3
4

d455 运行

可能没记全，因为是一开始忘了，全部复现完成后才重新记录的。

1. kimera_semantic_d455.launch
   复制一份Kimera_semantic_catkin_ws/src/Kimera-Semantics/kimera_semantics_ros/launch/kimera_semantics_euroc.launch改为kimera_semantic_d455.launch
   修改

  
  # 添加参数，传递给kimera_semantics_ros.launch
  
1
2
3

  
  
  
  
1
2
3
4

# 暂时是在复现，不需要点云，订阅了VIO发布的点云/stereo_gray/dense_stereo/pointcloud

1
2

2. kimera_semantics_d455.rviz
   复制一份Kimera_semantic_catkin_ws/src/Kimera-Semantics/kimera_semantics_ros/rviz/kimera_semantics_euroc.rviz改为kimera_semantics_d455.rviz
   后续可以在rviz中保存

运行

roslaunch kimera_vio_ros kimera_vio_ros_d455.launch run_stereo_dense:=true
roslaunch kimera_semantics_ros kimera_semantics_d455.launch
rviz -d $(rospack find kimera_semantics_ros)/rviz/kimera_semantics_d455.rviz
1
2
3

有mesh出来了，但是效果很差，可能是相机参数或者程序参数设置问题，后续再调

语义分割

目前使用的是deeplab_ros_pytorch

虚拟环境deeplab_ros
mkdir -p deeplab_ws/src
cd deeplab_ws
catkin init
cd src
git clone https://gitee.com/lu_ting_fei/ros-pytorch-deeplabv3.git

catkin config -DPYTHON_EXECUTABLE=/home/toniht/anaconda3/envs/deeplab_ros/bin/python -DPYTHON_INCLUDE_DIR=/home/toniht/anaconda3/envs/deeplab_ros/include/python3.8m -DPYTHON_LIBRARY=/home/toniht/anaconda3/envs/deeplab_ros/lib/libpython3.8.so -DCMAKE_BUILD_TYPE=Release -DSETUPTOOLS_DEB_LAYOUT=OFF
catkin_build
1
2
3
4
5
6
7
8
9

修改文件
deeplab_ws/src/ros-pytorch-deeplabv3/Img_seg_ros/predict.py

开始添加
sys.path.append('/home/toniht/workspace/deeplab_ws/src/ros-pytorch-deeplabv3/Img_seg_ros')

最后修改
img_topic=
ckpt = 

并修改
# 发布语义图像到ROS
ros_frame = bridge.cv2_to_imgmsg(seg_cv_img, "bgr8")
ros_frame.header.stamp = rospy.Time.now()
ros_frame.header.frame_id = "cam0" # 新增，方便后面发布语义点云时frame id一致
1
2
3
4
5
6
7
8
9
10
11
12

需要改为mask_rcnn

conda create -n mask_rcnn_ros python=3.7
conda activate mask_rcnn_ros
pip install boost protobuf scipy scikit-image ipython 'keras==2.4.3' opencv==3.4.10.35
pip install tensorflow-gpu==2.4.0
1
2
3
4

cd ~/catkin_ws/src
git clone https://github.com/ethz-asl/mask_rcnn_ros.git
git clone https://github.com/ethz-asl/depth_segmentation.git
git clone https://github.com/ethz-asl/opencv3_catkin.git
git clone https://github.com/ethz-asl/pcl_catkin.git

catkin build 			# 见 ERROR 2-7

# 如果找不到opencv或者pcl，删除重新编译
catkin build pcl_catkin opencv3_catkin
1
2
3
4
5
6
7
8
9
10

运行报错见ERROR 8、9

测试中

conda create -n mrcnn_ros python=3.8
conda activate mrcnn_ros

empy gdal boost pyyaml catkin_pkg
pip install boost protobuf scipy scikit-image ipython
pip install opencv-python==3.4.10.35

# tensorflow 1.15 https://flashgene.com/archives/242055.html
pip install nvidia-pyindex
pip install nvidia-tensorflow
1
2
3
4
5
6
7
8
9
10

catkin config -DPYTHON_EXECUTABLE=/home/toniht/anaconda3/envs/mrcnn_ros/bin/python -DPYTHON_INCLUDE_DIR=/home/toniht/anaconda3/envs/mrcnn_ros/include/python3.8 -DPYTHON_LIBRARY=/home/toniht/anaconda3/envs/mrcnn_ros/lib/libpython3.8.so -DCMAKE_BUILD_TYPE=Release -DSETUPTOOLS_DEB_LAYOUT=OFF

# 不改变c++路径的话会报错，但是忘记复制，应该是构建失败啥的来着
export CPLUS_INCLUDE_PATH=/home/toniht/anaconda3/envs/mrcnn_ros/include/python3.8:$CPLUS_INCLUDE_PATH

catkin build 
1
2
3
4
5
6

ERROR

ERROR 1：

Errors     << kimera_semantics_ros:make /home/toniht/workspace/Kimera_semantic_catkin_ws/logs/kimera_semantics_ros/build.make.000.log
In file included from /usr/include/pcl-1.10/pcl/pcl_macros.h:77,
                 from /usr/include/pcl-1.10/pcl/point_types.h:42,
                 from /home/toniht/workspace/Kimera_semantic_catkin_ws/src/Kimera-Semantics/kimera_semantics_ros/include/kimera_semantics_ros/semantic_simulation_server.h:8,
                 from /home/toniht/workspace/Kimera_semantic_catkin_ws/src/Kimera-Semantics/kimera_semantics_ros/src/semantic_simulation_server.cpp:1:
/usr/include/pcl-1.10/pcl/pcl_config.h:7:4: error: #error PCL requires C++14 or above
    7 |   #error PCL requires C++14 or above
1
2
3
4
5
6
7

FIX:
把对应的CMakeLists.txt文件中c++标准改为14(应该是两个kimera_semantics_ros和mesh)

set(CMAKE_CXX_STANDARD 14)
1

mesh中直接将c++11改为c++14就行

ERROR 2:

 Could not find a package configuration file provided by "catkin_simple"
  with any of the following names:

    catkin_simpleConfig.cmake
    catkin_simple-config.cmake
1
2
3
4
5

FIX:

cd src
git clone https://github.com/catkin/catkin_simple
1
2

ERROR 3：

Errors     << pcl_catkin:cmake /home/toniht/workspace/mask_rcnn_catkin_ws/logs/pcl_catkin/build.cmake.000.log                                                                                           
CMake Error at /home/toniht/workspace/mask_rcnn_catkin_ws/src/pcl_catkin/pcl_catkin/CMakeLists.txt:15 (MESSAGE):
  CMAKE_BUILD_TYPE must not be empty!
1
2
3

FIX:

catkin clean -y
catkin config -DCMAKE_BUILD_TYPE=Release
1
2

ERROR 4：

/home/toniht/anaconda3/envs/mask_rcnn_ros/include/boost/python/detail/wrap_python.hpp:57:11: fatal error: pyconfig.h: 没有那个文件或目录
   57 | # include 
1
2

FIX:

export CPLUS_INCLUDE_PATH=/home/toniht/anaconda3/envs/mask_rcnn_ros/include/python3.7m:$CPLUS_INCLUDE_PATH
1

ERROR 5：

/home/toniht/anaconda3/envs/mask_rcnn_ros/lib/python3.7/site-packages/numpy/core/include/numpy/__multiarray_api.h:1532:151: error: return-statement with a value, in function returning ‘void’ [-fpermissive]
 1532 | #define import_array() {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import"); return NULL; } }
1
2

FIX:
import_array()报错，返回值类型与函数类型不匹配

ERROR 6：

/home/toniht/anaconda3/envs/mask_rcnn_ros/include/boost/bind.hpp:36:1: note: in expansion of macro ‘BOOST_PRAGMA_MESSAGE’
   36 | BOOST_PRAGMA_MESSAGE(
      | ^~~~~~~~~~~~~~~~~~~~
/usr/bin/ld: 找不到 -lBoost::system
/usr/bin/ld: 找不到 -lBoost::filesystem
/usr/bin/ld: 找不到 -lBoost::thread
collect2: error: ld returned 1 exit status
1
2
3
4
5
6
7

FIX:
在depth_segmentation的CMakeLists.txt中添加

# 设置虚拟环境路径
set(VIRTUAL_ENV_PATH "/home/toniht/anaconda3/envs/mask_rcnn_ros")

# 设置 Boost 头文件和库的路径
set(Boost_INCLUDE_DIR "${VIRTUAL_ENV_PATH}/include")
set(Boost_LIBRARY_DIR "${VIRTUAL_ENV_PATH}/lib")

find_package(Boost COMPONENTS system filesystem thread REQUIRED)

include_directories(${Boost_INCLUDE_DIRS})

link_directories(${Boost_LIBRAYY_DIRS})
1
2
3
4
5
6
7
8
9
10
11
12

ERROR 7：

/usr/bin/ld: /home/toniht/workspace/kimera_mask_rcnn_ws/devel/lib/libkimera_semantics_ros.so: undefined reference to `cv::String::deallocate()'
1

FIX:
ubuntu20.04+opencv4.2.0编译kimera
在kimera_semantics_ros的CMakeLists.txt中加入

find_package(opencv3_catkin REQUIRED)
# 在最后面加
target_link_libraries(${PROJECT_NAME} ${opencv3_catkin_LIBRARIES}) 
1
2
3

ERROR 8:

'module' object has no attribute 'placeholder'
1

FIX：
tensorflow2.x和tensorflow1.x的差异
修改两个文件Kimera_semantic_catkin_ws/src/mask_rcnn_ros/src/mask_rcnn_ros/model.py和/home/toniht/anaconda3/envs/mask_rcnn_ros/lib/python3.7/site-packages/keras/backend/tensorflow_backend.py
将其中的

import tensorflow as tf
-> import tensorflow.compat.v1 as tf
tf.disable_v2_behavior()
1
2
3

ERROR 9：

  File "/home/toniht/anaconda3/envs/mask_rcnn_ros/lib/python3.7/site-packages/keras/engine/topology.py", line 3418, in load_weights_from_hdf5_group_by_name
    original_keras_version = f.attrs['keras_version'].decode('utf8')
AttributeError: 'str' object has no attribute 'decode'
1
2
3

FIX:

pip install 'h5py<3.0.0'
1

ERROR 10:

  File "/home/toniht/workspace/kimera_mask_rcnn_ws/src/mask_rcnn_ros/src/mask_rcnn_ros/utils.py", line 420, in resize_image
    image = scipy.misc.imresize(
AttributeError: module 'scipy.misc' has no attribute 'imresize'

1
2
3
4

FIX:
修改报错文件utils.py

# 添加
from skimage.transform import resize
# 修改报错语句
image = resize(
            image, (int(round(h * scale)), int(round(w * scale))))
1
2
3
4
5

ERROR 11:

/home/toniht/anaconda3/envs/mrcnn_ros/lib/python3.8/site-packages/numpy/core/include/numpy/__multiarray_api.h:1553:151: error: return-statement with a value, in function returning ‘void’ [-fpermissive]
 1553 | #define import_array() {if (_import_array() < 0) {PyErr_Print(); PyErr_SetString(PyExc_ImportError, "numpy.core.multiarray failed to import"); return NULL; } }
1
2

FIX:
修改"/home/toniht/anaconda3/envs/mrcnn_ros/lib/python3.8/site-packages/numpy/core/include/numpy/__multiarray_api.h"
删去对应的NULL，重新编译