• bclinux aarch64 ceph 14.2.10 对象存储 http网关 CEPH OBJECT GATEWAY Civetweb

    相关内容

    bclinux aarch64 ceph 14.2.10 文件存储 Ceph File System, 需要部署mds： ceph-deploy mds-CSDN博客

    ceph-deploy bclinux aarch64 ceph 14.2.10【3】vdbench fsd 文件系统测试-CSDN博客

    ceph-deploy bclinux aarch64 ceph 14.2.10【2】vdbench rbd 块设备rbd 测试失败-CSDN博客

    ceph-deploy bclinux aarch64 ceph 14.2.10-CSDN博客

    部署对象存储网关

    1. cd /etc/ceph
    2. ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3

    成功日志

    1. [root@ceph-0 ceph]# ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3
    2. [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
    3. [ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3
    4. [ceph_deploy.cli][INFO  ] ceph-deploy options:
    5. [ceph_deploy.cli][INFO  ]  username                      : None
    6. [ceph_deploy.cli][INFO  ]  verbose                       : False
    7. [ceph_deploy.cli][INFO  ]  rgw                           : [('ceph-0', 'rgw.ceph-0'), ('ceph-1', 'rgw.ceph-1'), ('ceph-2', 'rgw.ceph-2'), ('ceph-3', 'rgw.ceph-3')]
    8. [ceph_deploy.cli][INFO  ]  overwrite_conf                : False
    9. [ceph_deploy.cli][INFO  ]  subcommand                    : create
    10. [ceph_deploy.cli][INFO  ]  quiet                         : False
    11. [ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0xffffa1b8faa0>
    12. [ceph_deploy.cli][INFO  ]  cluster                       : ceph
    13. [ceph_deploy.cli][INFO  ]  func                          : <function rgw at 0xffffa1d554d0>
    14. [ceph_deploy.cli][INFO  ]  ceph_conf                     : None
    15. [ceph_deploy.cli][INFO  ]  default_release               : False
    16. [ceph_deploy.rgw][DEBUG ] Deploying rgw, cluster ceph hosts ceph-0:rgw.ceph-0 ceph-1:rgw.ceph-1 ceph-2:rgw.ceph-2 ceph-3:rgw.ceph-3
    17. [ceph-0][DEBUG ] connected to host: ceph-0 
    18. [ceph-0][DEBUG ] detect platform information from remote host
    19. 21.10U3 LTS
    20. bclinux
    21. [ceph-0][DEBUG ] detect machine type
    22. [ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
    23. [ceph_deploy.rgw][DEBUG ] remote host will use systemd
    24. [ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-0
    25. [ceph-0][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
    26. [ceph-0][WARNIN] rgw keyring does not exist yet, creating one
    27. [ceph-0][DEBUG ] create a keyring file
    28. [ceph-0][DEBUG ] create path recursively if it doesn't exist
    29. [ceph-0][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-0 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-0/keyring
    30. [ceph-0][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-0
    31. [ceph-0][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-0.service → /usr/lib/systemd/system/ceph-radosgw@.service.
    32. [ceph-0][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-0
    33. [ceph-0][INFO  ] Running command: systemctl enable ceph.target
    34. [ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-0 and default port 7480
    35. dhclient(1787) is already running - exiting. 
    37. This version of ISC DHCP is based on the release available
    38. on ftp.isc.org. Features have been added and other changes
    39. have been made to the base software release in order to make
    40. it work better with this distribution.
    42. Please report issues with this software via: 
    43. https://gitee.com/src-openeuler/dhcp/issues
    45. exiting.
    46. dhclient(1787) is already running - exiting. 
    48. This version of ISC DHCP is based on the release available
    49. on ftp.isc.org. Features have been added and other changes
    50. have been made to the base software release in order to make
    51. it work better with this distribution.
    53. Please report issues with this software via: 
    54. https://gitee.com/src-openeuler/dhcp/issues
    56. exiting.
    57. [ceph-1][DEBUG ] connected to host: ceph-1 
    58. [ceph-1][DEBUG ] detect platform information from remote host
    59. 21.10U3 LTS
    60. bclinux
    61. [ceph-1][DEBUG ] detect machine type
    62. [ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
    63. [ceph_deploy.rgw][DEBUG ] remote host will use systemd
    64. [ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-1
    65. [ceph-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
    66. [ceph-1][WARNIN] rgw keyring does not exist yet, creating one
    67. [ceph-1][DEBUG ] create a keyring file
    68. [ceph-1][DEBUG ] create path recursively if it doesn't exist
    69. [ceph-1][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-1 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-1/keyring
    70. [ceph-1][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-1
    71. [ceph-1][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-1.service → /usr/lib/systemd/system/ceph-radosgw@.service.
    72. [ceph-1][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-1
    73. [ceph-1][INFO  ] Running command: systemctl enable ceph.target
    74. [ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-1 and default port 7480
    75. dhclient(1742) is already running - exiting. 
    76.  
    77. This version of ISC DHCP is based on the release available
    78. on ftp.isc.org. Features have been added and other changes
    79. have been made to the base software release in order to make
    80. it work better with this distribution.
    81.  
    82. Please report issues with this software via: 
    83. https://gitee.com/src-openeuler/dhcp/issues
    84.  
    85. exiting.
    86. dhclient(1742) is already running - exiting. 
    87.  
    88. This version of ISC DHCP is based on the release available
    89. on ftp.isc.org. Features have been added and other changes
    90. have been made to the base software release in order to make
    91. it work better with this distribution.
    92.  
    93. Please report issues with this software via: 
    94. https://gitee.com/src-openeuler/dhcp/issues
    95.  
    96. exiting.
    97. [ceph-2][DEBUG ] connected to host: ceph-2 
    98. [ceph-2][DEBUG ] detect platform information from remote host
    99. 21.10U3 LTS
    100. bclinux
    101. [ceph-2][DEBUG ] detect machine type
    102. [ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
    103. [ceph_deploy.rgw][DEBUG ] remote host will use systemd
    104. [ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-2
    105. [ceph-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
    106. [ceph-2][WARNIN] rgw keyring does not exist yet, creating one
    107. [ceph-2][DEBUG ] create a keyring file
    108. [ceph-2][DEBUG ] create path recursively if it doesn't exist
    109. [ceph-2][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-2 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-2/keyring
    110. [ceph-2][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-2
    111. [ceph-2][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-2.service → /usr/lib/systemd/system/ceph-radosgw@.service.
    112. [ceph-2][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-2
    113. [ceph-2][INFO  ] Running command: systemctl enable ceph.target
    114. [ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-2 and default port 7480
    115. dhclient(1722) is already running - exiting. 
    117. This version of ISC DHCP is based on the release available
    118. on ftp.isc.org. Features have been added and other changes
    119. have been made to the base software release in order to make
    120. it work better with this distribution.
    122. Please report issues with this software via: 
    123. https://gitee.com/src-openeuler/dhcp/issues
    125. exiting.
    126. dhclient(1722) is already running - exiting. 
    128. This version of ISC DHCP is based on the release available
    129. on ftp.isc.org. Features have been added and other changes
    130. have been made to the base software release in order to make
    131. it work better with this distribution.
    133. Please report issues with this software via: 
    134. https://gitee.com/src-openeuler/dhcp/issues
    136. exiting.
    137. [ceph-3][DEBUG ] connected to host: ceph-3 
    138. [ceph-3][DEBUG ] detect platform information from remote host
    139. 21.10U3 LTS
    140. bclinux
    141. [ceph-3][DEBUG ] detect machine type
    142. [ceph_deploy.rgw][INFO  ] Distro info: bclinux 21.10U3 21.10U3 LTS
    143. [ceph_deploy.rgw][DEBUG ] remote host will use systemd
    144. [ceph_deploy.rgw][DEBUG ] deploying rgw bootstrap to ceph-3
    145. [ceph-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
    146. [ceph-3][WARNIN] rgw keyring does not exist yet, creating one
    147. [ceph-3][DEBUG ] create a keyring file
    148. [ceph-3][DEBUG ] create path recursively if it doesn't exist
    149. [ceph-3][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-rgw --keyring /var/lib/ceph/bootstrap-rgw/ceph.keyring auth get-or-create client.rgw.ceph-3 osd allow rwx mon allow rw -o /var/lib/ceph/radosgw/ceph-rgw.ceph-3/keyring
    150. [ceph-3][INFO  ] Running command: systemctl enable ceph-radosgw@rgw.ceph-3
    151. [ceph-3][WARNIN] Created symlink /etc/systemd/system/ceph-radosgw.target.wants/ceph-radosgw@rgw.ceph-3.service → /usr/lib/systemd/system/ceph-radosgw@.service.
    152. [ceph-3][INFO  ] Running command: systemctl start ceph-radosgw@rgw.ceph-3
    153. [ceph-3][INFO  ] Running command: systemctl enable ceph.target
    154. [ceph_deploy.rgw][INFO  ] The Ceph Object Gateway (RGW) is now running on host ceph-3 and default port 7480
    155. [root@ceph-0 ceph]# cp: cannot stat '/tmp/security_updates': No such file or directory
    156. cp: cannot stat '/tmp/security_updates': No such file or directory
    157. cp: cannot stat '/tmp/security_updates': No such file or directory

    创建成功

    验证服务状态（失败）

    日志目录

    /var/log/ceph

    日志文件

    ceph-client.rgw.ceph-0.log

    清空日志文件，重启服务

    1. cd /var/log/ceph
    2. echo "" > ceph-client.rgw.ceph-0.log
    3. systemctl start ceph-radosgw@rgw-ceph-0

    先看服务日志

    journalctl -xeu ceph-radosgw@rgw-ceph-0

    日志

    Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory
    Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 AuthRegistry(0xffff7a330148) no keyring found at /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring, disabling cephx
    Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory
    Nov 14 13:38:08 ceph-0 radosgw[58375]: 2023-11-14 13:38:08.450 ffff7ab37010 -1 AuthRegistry(0xffffdc7f98c8) no keyring found at /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring, disabling cephx
    Nov 14 13:38:08 ceph-0 radosgw[58375]: failed to fetch mon config (--no-mon-config to skip)
    Nov 14 13:38:08 ceph-0 systemd[1]: ceph-radosgw@rgw-ceph-0.service: Main process exited, code=exited, status=1/FAILURE
     

    没有按照官方文档执行网关安装有关？ceph-deploy install --rgw node..

    但是这个版本的ceph-deploy指定的ceph版本太低，且是el6

    解决  /var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring: (2) No such file or directory

    查看/etc/ceph目录下的keyring文件，发现已生成了ceph.bootstrap-rgw.keyring

    错误的路径/var/lib/ceph/radosgw/ceph-rgw-ceph-0/keyring

    正确的路径/var/lib/ceph/radosgw/ceph-rgw.ceph-0/keyring

    不知道为什么路径有一个符号错误

    做个软连接

    1. cd /var/lib/ceph/radosgw
    2. ln -s ceph-rgw.ceph-0 ceph-rgw-ceph-0

    错误：

    Nov 14 14:11:43 ceph-0 radosgw[59264]: 2023-11-14 14:11:43.076 ffff99055730 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
    Nov 14 14:11:43 ceph-0 radosgw[59264]: 2023-11-14 14:11:43.080 ffff99856730 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
    Nov 14 14:11:43 ceph-0 radosgw[59264]: failed to fetch mon config (--no-mon-config to skip)
    Nov 14 14:11:43 ceph-0 systemd[1]: ceph-radosgw@rgw-ceph-0.service: Main process exited, code=exited, status=1/FAILURE

    重新执行部署

    1. cd /etc/ceph
    2. ceph-deploy rgw create ceph-0 ceph-1 ceph-2 ceph-3

    重新部署后，会生成/var/log/ceph/ceph-client.rgw.ceph-0.log日志文件

    其中有日志

    Numerical result out of range

    2023-11-14 14:32:24.461 ffff8abe8010  0 deferred set uid:gid to 1000:1000 (ceph:ceph)

    2023-11-14 14:32:24.461 ffff8abe8010  0 ceph version 14.2.10 (b340acf629a010a74d90da5782a2c5fe0b54ac20) nautilus (stable), process radosgw, pid 60035

    2023-11-14 14:32:25.509 ffff8abe8010  0 rgw_init_ioctx ERROR: librados::Rados::pool_create returned (34) Numerical result out of range (this can be due to a pool or placement group misconfiguration, e.g. pg_num < pgp_num or mon_max_pg_per_osd exceeded)

    2023-11-14 14:32:25.509 ffff8abe8010  0 failed reading realm info: ret -34 (34) Numerical result out of range

    2023-11-14 14:32:25.509 ffff8abe8010  0 ERROR: failed to start notify service ((34) Numerical result out of range

    2023-11-14 14:32:25.509 ffff8abe8010  0 ERROR: failed to init services (ret=(34) Numerical result out of range)

    2023-11-14 14:32:25.513 ffff8abe8010 -1 Couldn't init storage provider (RADOS)

    设置默认pg_num pgs_num

    /etc/ceph/ceph.conf

    1. osd pool default size = 3
    2. osd pool default min_size = 1
    3. osd pool default pg num = 128
    4. osd pool default pgp num = 128

    重新部署(--overwrite-conf)

    1. cd /etc/ceph
    2. ceph-deploy --overwrite-conf rgw create ceph-0 ceph-1 ceph-2 ceph-3

    故障依旧

    成功的方案

    删掉所有pool，重新部署，参考以下历史记录

     652  ceph osd pool ls
      653  ceph osd pool rm vdbench
      654  ceph osd pool rm vdbench vdbench --yes-i-really-really-mean-it
      655  ls /dev/
      656  mount
      657  df -h
      658  umount /mnt/rbd0
      659  ceph osd ls
      660  ceph osd pool ls
      661  ceph osd volume ls
      662  ceph osd dump
      663  ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it
      664  ceph fs ls
      665  ceph fs rm cephfs
      666  ceph fs fail
      667  ceph fs fail cephfs
      668  ceph -s
      669  ceph fs status cephfs
      670  ceph fs ls
      671  ceph fs rm cephfs
      672  ceph fs rm cephfs --yes-i-really-mean-it
      673  ceph fs ls
      674  ceph osd pool ls
      675  ceph osd pool rm cephfs_data cephfs_data  --yes-i-really-mean-it
      676  ceph osd pool rm cephfs_data cephfs_data  --yes-i-really-really-mean-it
      677  ceph osd pool rm cephfs_metadata cephfs_metadata  --yes-i-really-really-mean-it
      678  ceph -
      679  ceph -s
      680  cd /etc/ceph
      681  ceph-deploy rgw create ceph-0 
      682  cat /var/log/ceph/ceph-client.rgw.ceph-0.log 
      683  systemctl status ceph-radosgw@rgw.ceph-0
    再来一次集群环境的部署

    1. cd /etc/ceph
    2. ceph-deploy --overwrite-conf rgw create ceph-0 ceph-1 ceph-2 ceph-3

    部分日志

    ceph -s rgw已经起来了

    ======================

    以下为参考官方文档的失败经理，留档参考

    部署对象存储网关rgw（失败）

    ceph-deploy install --rgw ceph-0 ceph-1 ceph-2 ceph-3

    故障 Unable to find a match: epel-release

    dnf install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm -y

    故障Status code: 404

    坑爹的阿里云镜像，没有epel 8，bclinux又是el8版本！

    采用huawei epel镜像

    1. [epel-everything]
    2. name=huawei epel mirror
    3. baseurl=https://repo.huaweicloud.com/epel/8/Everything/aarch64/
    4. enabled=1
    5. gpgcheck=0
    yum install epel-release --nobest --skip-broken

    ceph-deploy 会强制执行yum install epel-release(失败)

    注释掉epel-release的安装

    修改/usr/lib/python2.7/site-packages/ceph_deploy/hosts/centos/install.py 60行，注释掉epel-release的安装（前面已经手动--skip-broken安装好了）

    生效了

    故障 Unable to find a match: yum-plugin-priorities

    参考上面思路，直接注释掉/usr/lib/python2.7/site-packages/ceph_deploy/hosts/centos/install.py 61,62行

    自动找到老版本ceph-release-1-0.el6 Status code: 404

    手动安装

    修改/usr/lib/python2.7/site-packages/ceph_deploy/install.py

    安装版本参数args.release = 'mimic' 改为 args.release = 'nautilus

    关于python3版本

    bclinux python3.7.9

    cenos7 python3.6.8

    openeuler 20.03 lts sp3 python 3.7.9

    参考

    Ceph Object Gateway — Ceph Documentation

    如何在 CentOS 8 和 RHEL 8 服务器上启用 EPEL 仓库 - 知乎 (zhihu.com)

    ceph-deploy部署ceph-nautilus_ceph nautilus-CSDN博客

  • 相关阅读:
    Linux 内核页表管理
    解决mysql中group_concat长度限制的方案
    蓝桥楼赛第30期-Python模块大比拼-第一天赛题 题解
    排序算法-快速排序法（QuickSort）
    一次k8s docker下.net程序的异常行为dump诊断
    Unity中实现溶解(Dissolve)特效及其原理解析
    Transformer架构
    Septentrio接收机二进制的BDS b2b改正数解码
    基于Surprise协同过滤实现短视频推荐
    如何恢复edge的自动翻译功能
  • 原文地址：https://blog.csdn.net/hknaruto/article/details/134391227