• harbor v1.7.1镜像仓库无法访问，并提示502 Bad Gateway

    一、背景：

    在巡检rancher平台时发现有一个服务运行报错了，查看该服务容器事件时提示连接不到harbor镜像仓库。

    二、harbor镜像仓库访问问题分析过程： 

    1、确认harbor仓库是否可以访问：

    发现无法访问时，第一时间是通过浏览器去访问harbor仓库是否能正常访问：

    http://harbor.jx.shu.com

    发现无法访问，然后登入到对应的harbor服务器上去查看harbor服务是否正常。

    2、查看分析harbor服务器的网络情况： 

    通过堡垒机去访问harbor服务器时发现无法登入上去，这时候就需要找硬件基础工程师进行处理了。 

    3、查看harbor服务运行状态： 

    硬件工程师处理好harbor服务器无法登入的问题，之后登入到harbor服务器上去，并通过docker-compose命令查看服务运行状态，如下：

    1. root@harbor:/home/service/harbor# docker-compose ps
    2.        Name                     Command                 State                                   Ports                              
    3. -----------------------------------------------------------------------------------------------------------------------------------
    4. harbor-adminserver   /harbor/start.sh                 Up                                                                           
    5. harbor-core          /harbor/start.sh                 Up                                                                           
    6. harbor-db            /entrypoint.sh postgres          Up           5432/tcp                                                        
    7. harbor-jobservice    /harbor/start.sh                 Up                                                                           
    8. harbor-log           /bin/sh -c /usr/local/bin/ ...   Up           127.0.0.1:1514->10514/tcp                                       
    9. harbor-portal        nginx -g daemon off;             Restarting                                                                   
    10. nginx                nginx -g daemon off;             Up           0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp, 0.0.0.0:80->80/tcp
    11. redis                docker-entrypoint.sh redis ...   Up           6379/tcp                                                        
    12. registry             /entrypoint.sh /etc/regist ...   Up           5000/tcp                                                        
    13. registryctl          /harbor/start.sh                 Up

    根据查询到情况，可以发现harbor-portal容器服务运行异常，然后查看harbor对应的yaml文件内容：

    1. version: '2'
    2. services:
    3.   log:
    4.     image: goharbor/harbor-log:v1.7.1
    5.     container_name: harbor-log 
    6.     restart: always
    7.     dns_search: .
    8.     cap_drop:
    9.       - ALL
    10.     cap_add:
    11.       - CHOWN
    12.       - DAC_OVERRIDE
    13.       - SETGID
    14.       - SETUID
    15.     volumes:
    16.       - /var/log/harbor/:/var/log/docker/:z
    17.       - ./common/config/log/:/etc/logrotate.d/:z
    18.     ports:
    19.       - 127.0.0.1:1514:10514
    20.     networks:
    21.       - harbor
    22.   registry:
    23.     image: goharbor/registry-photon:v2.6.2-v1.7.1
    24.     container_name: registry
    25.     restart: always
    26.     cap_drop:
    27.       - ALL
    28.     cap_add:
    29.       - CHOWN
    30.       - SETGID
    31.       - SETUID
    32.     volumes:
    33.       - /data/registry:/storage:z
    34.       - ./common/config/registry/:/etc/registry/:z
    35.       - ./common/config/custom-ca-bundle.crt:/harbor_cust_cert/custom-ca-bundle.crt:z
    36.     networks:
    37.       - harbor
    38.     dns_search: .
    39.     depends_on:
    40.       - log
    41.     logging:
    42.       driver: "syslog"
    43.       options:  
    44.         syslog-address: "tcp://127.0.0.1:1514"
    45.         tag: "registry"
    46.   registryctl:
    47.     image: goharbor/harbor-registryctl:v1.7.1
    48.     container_name: registryctl
    49.     env_file:
    50.       - ./common/config/registryctl/env
    51.     restart: always
    52.     cap_drop:
    53.       - ALL
    54.     cap_add:
    55.       - CHOWN
    56.       - SETGID
    57.       - SETUID
    58.     volumes:
    59.       - /data/registry:/storage:z
    60.       - ./common/config/registry/:/etc/registry/:z
    61.       - ./common/config/registryctl/config.yml:/etc/registryctl/config.yml:z
    62.     networks:
    63.       - harbor
    64.     dns_search: .
    65.     depends_on:
    66.       - log
    67.     logging:
    68.       driver: "syslog"
    69.       options:  
    70.         syslog-address: "tcp://127.0.0.1:1514"
    71.         tag: "registryctl"
    72.   postgresql:
    73.     image: goharbor/harbor-db:v1.7.1
    74.     container_name: harbor-db
    75.     restart: always
    76.     cap_drop:
    77.       - ALL
    78.     cap_add:
    79.       - CHOWN
    80.       - DAC_OVERRIDE
    81.       - SETGID
    82.       - SETUID
    83.     volumes:
    84.       - /data/database:/var/lib/postgresql/data:z
    85.     networks:
    86.       - harbor
    87.     dns_search: .
    88.     env_file:
    89.       - ./common/config/db/env
    90.     depends_on:
    91.       - log
    92.     logging:
    93.       driver: "syslog"
    94.       options:  
    95.         syslog-address: "tcp://127.0.0.1:1514"
    96.         tag: "postgresql"
    97.   adminserver:
    98.     image: goharbor/harbor-adminserver:v1.7.1
    99.     container_name: harbor-adminserver
    100.     env_file:
    101.       - ./common/config/adminserver/env
    102.     restart: always
    103.     cap_drop:
    104.       - ALL
    105.     cap_add:
    106.       - CHOWN
    107.       - SETGID
    108.       - SETUID
    109.     volumes:
    110.       - /data/config/:/etc/adminserver/config/:z
    111.       - /data/secretkey:/etc/adminserver/key:z
    112.       - /data/:/data/:z
    113.     networks:
    114.       - harbor
    115.     dns_search: .
    116.     depends_on:
    117.       - log
    118.     logging:
    119.       driver: "syslog"
    120.       options:  
    121.         syslog-address: "tcp://127.0.0.1:1514"
    122.         tag: "adminserver"
    123.   core:
    124.     image: goharbor/harbor-core:v1.7.1
    125.     container_name: harbor-core
    126.     env_file:
    127.       - ./common/config/core/env
    128.     restart: always
    129.     cap_drop:
    130.       - ALL
    131.     cap_add:
    132.       - SETGID
    133.       - SETUID
    134.     volumes:
    135.       - ./common/config/core/app.conf:/etc/core/app.conf:z
    136.       - ./common/config/core/private_key.pem:/etc/core/private_key.pem:z
    137.       - ./common/config/core/certificates/:/etc/core/certificates/:z
    138.       - /data/secretkey:/etc/core/key:z
    139.       - /data/ca_download/:/etc/core/ca/:z
    140.       - /data/psc/:/etc/core/token/:z
    141.       - /data/:/data/:z
    142.     networks:
    143.       - harbor
    144.     dns_search: .
    145.     depends_on:
    146.       - log
    147.       - adminserver
    148.       - registry
    149.     logging:
    150.       driver: "syslog"
    151.       options:  
    152.         syslog-address: "tcp://127.0.0.1:1514"
    153.         tag: "core"
    154.   portal:
    155.     image: goharbor/harbor-portal:v1.7.1
    156.     container_name: harbor-portal
    157.     restart: always
    158.     cap_drop:
    159.       - ALL
    160.     cap_add:
    161.       - CHOWN
    162.       - SETGID
    163.       - SETUID
    164.       - NET_BIND_SERVICE
    165.     networks:
    166.       - harbor
    167.     dns_search: .
    168.     depends_on:
    169.       - log
    170.       - core
    171.     logging:
    172.       driver: "syslog"
    173.       options:
    174.         syslog-address: "tcp://127.0.0.1:1514"
    175.         tag: "portal"
    176.  
    177.   jobservice:
    178.     image: goharbor/harbor-jobservice:v1.7.1
    179.     container_name: harbor-jobservice
    180.     env_file:
    181.       - ./common/config/jobservice/env
    182.     restart: always
    183.     cap_drop:
    184.       - ALL
    185.     cap_add:
    186.       - CHOWN
    187.       - SETGID
    188.       - SETUID
    189.     volumes:
    190.       - /data/job_logs:/var/log/jobs:z
    191.       - ./common/config/jobservice/config.yml:/etc/jobservice/config.yml:z
    192.     networks:
    193.       - harbor
    194.     dns_search: .
    195.     depends_on:
    196.       - redis
    197.       - core
    198.       - adminserver
    199.     logging:
    200.       driver: "syslog"
    201.       options:
    202.         syslog-address: "tcp://127.0.0.1:1514"
    203.         tag: "jobservice"
    204.   redis:
    205.     image: goharbor/redis-photon:v1.7.1
    206.     container_name: redis
    207.     restart: always
    208.     cap_drop:
    209.       - ALL
    210.     cap_add:
    211.       - CHOWN
    212.       - SETGID
    213.       - SETUID
    214.     volumes:
    215.       - /data/redis:/var/lib/redis
    216.     networks:
    217.       - harbor
    218.     dns_search: .
    219.     depends_on:
    220.       - log
    221.     logging:
    222.       driver: "syslog"
    223.       options:
    224.         syslog-address: "tcp://127.0.0.1:1514"
    225.         tag: "redis"
    226.   proxy:
    227.     image: goharbor/nginx-photon:v1.7.1
    228.     container_name: nginx
    229.     restart: always
    230.     cap_drop:
    231.       - ALL
    232.     cap_add:
    233.       - CHOWN
    234.       - SETGID
    235.       - SETUID
    236.       - NET_BIND_SERVICE
    237.     volumes:
    238.       - ./common/config/nginx:/etc/nginx:z
    239.     networks:
    240.       - harbor
    241.     dns_search: .
    242.     ports:
    243.       - 80:80
    244.       - 443:443
    245.       - 4443:4443
    246.     depends_on:
    247.       - postgresql
    248.       - registry
    249.       - core
    250.       - portal
    251.       - log
    252.     logging:
    253.       driver: "syslog"
    254.       options:  
    255.         syslog-address: "tcp://127.0.0.1:1514"
    256.         tag: "proxy"
    257. networks:
    258.   harbor:
    259.     external: false

    发现harbor-portal容器服务的日志是存放在/var/log/harbor目录下的，需要到该目录找对应服务的日志内容：

    1. root@harbor:/home/service/harbor# cd /var/log/harbor/
    2. root@harbor:/var/log/harbor# ls
    3. adminserver.log  dev-198-1b9b616909af44f90ae1566674c19032ec13c8da.log  portal.log      proxy.log  registryctl.log
    4. core.log         jobservice.log                                        postgresql.log  redis.log  registry.log

    root@harbor:/var/log/harbor# tail -100f portal.log

    查看的日志报错如下：

    1. ........
    2. Feb 18 14:43:15 localhost portal[97624]: 2024/02/18 06:43:15 [emerg] 1#0: mkdir() "/etc//nginx/client_body_temp" failed (13: Permission denied)
    3. Feb 18 14:43:15 localhost portal[97624]: nginx: [emerg] mkdir() "/etc//nginx/client_body_temp" failed (13: Permission denied)
    4. Feb 18 14:43:41 localhost portal[97624]: 2024/02/18 06:43:41 [emerg] 1#0: mkdir() "/etc//nginx/client_body_temp" failed (13: Permission denied)
    5. Feb 18 14:43:41 localhost portal[97624]: nginx: [emerg] mkdir() "/etc//nginx/client_body_temp" failed (13: Permission denied)
    6. Feb 18 14:44:32 172.18.0.1 portal[97624]: 2024/02/18 06:44:32 [emerg] 1#0: mkdir() "/etc//nginx/client_body_temp" failed (13: Permission denied)
    7. Feb 18 14:44:32 172.18.0.1 portal[97624]: nginx: [emerg] mkdir() "/etc//nginx/client_body_temp" failed (13: Permission denied)
    8. Feb 18 14:46:15 172.18.0.1 portal[97624]: 2024/02/18 06:46:15 [emerg] 1#0: mkdir() "/etc//nginx/client_body_temp" failed (13: Permission denied)
    9. Feb 18 14:46:15 172.18.0.1 portal[97624]: nginx: [emerg] mkdir() "/etc//nginx/client_body_temp" failed (13: Permission denied)
    10. Feb 18 14:49:40 localhost portal[97624]: 2024/02/18 06:49:40 [emerg] 1#0: mkdir() "/etc//nginx/client_body_temp" failed (13: Permission denied)
    11. Feb 18 14:49:40 localhost portal[97624]: nginx: [emerg] mkdir() "/etc//nginx/client_body_temp" failed (13: Permission denied)

    但是该服务运行的不是nginx服务的，所以日志中报错的内容，提供不了排查方向。

    4、解决harbor服务运行异常问题： 

    首先想到harbor服务器有重启过，是否有可能因为docker的网络有问题导致的，需要重启一下docker服务，然后再去重启一下harbor服务的。

    1. root@harbor:/home/service/harbor# docker-compose stop
    2. Stopping nginx              ... done
    3. Stopping harbor-jobservice  ... done
    4. Stopping harbor-portal      ... done
    5. Stopping harbor-core        ... done
    6. Stopping redis              ... done
    7. Stopping registryctl        ... done
    8. Stopping registry           ... done
    9. Stopping harbor-db          ... done
    10. Stopping harbor-adminserver ... done
    11. Stopping harbor-log         ... done
    12.  
    13. root@harbor:/home/service/harbor# systemctl restart docker
    14.  
    15. root@harbor:/home/service/harbor# docker-compose start
    16. Starting log         ... done
    17. Starting postgresql  ... done
    18. Starting redis       ... done
    19. Starting adminserver ... done
    20. Starting registry    ... done
    21. Starting core        ... done
    22. Starting jobservice  ... done
    23. Starting portal      ... done
    24. Starting proxy       ... done
    25. Starting registryctl ... done
    26.  
    27. root@harbor:/home/service/harbor# docker-compose ps
    28.        Name                     Command                 State                                   Ports                              
    29. -----------------------------------------------------------------------------------------------------------------------------------
    30. harbor-adminserver   /harbor/start.sh                 Up                                                                           
    31. harbor-core          /harbor/start.sh                 Up                                                                           
    32. harbor-db            /entrypoint.sh postgres          Up           5432/tcp                                                        
    33. harbor-jobservice    /harbor/start.sh                 Up                                                                           
    34. harbor-log           /bin/sh -c /usr/local/bin/ ...   Up           127.0.0.1:1514->10514/tcp                                       
    35. harbor-portal        nginx -g daemon off;             Restarting                                                                   
    36. nginx                nginx -g daemon off;             Up           0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp, 0.0.0.0:80->80/tcp
    37. redis                docker-entrypoint.sh redis ...   Up           6379/tcp                                                        
    38. registry             /entrypoint.sh /etc/regist ...   Up           5000/tcp                                                        
    39. registryctl          /harbor/start.sh                 Up

    然后发现还是不行，网络查找可以尝试通过install.sh脚本的访问处理，就尝试了一下：

    1. root@harbor:/home/service/harbor# ./install.sh 
    2.  
    3. [Step 0]: checking installation environment ...
    4.  
    5. Note: docker version: 17.03.2
    6.  
    7. Note: docker-compose version: 1.18.0
    8.  
    9.  
    10. [Step 1]: preparing environment ...
    11. Clearing the configuration file: ./common/config/registryctl/env
    12. Clearing the configuration file: ./common/config/registryctl/config.yml
    13. Clearing the configuration file: ./common/config/db/env
    14. Clearing the configuration file: ./common/config/core/env
    15. Clearing the configuration file: ./common/config/core/app.conf
    16. Clearing the configuration file: ./common/config/core/private_key.pem
    17. Clearing the configuration file: ./common/config/log/logrotate.conf
    18. Clearing the configuration file: ./common/config/adminserver/env
    19. Clearing the configuration file: ./common/config/registry/config.yml
    20. Clearing the configuration file: ./common/config/registry/root.crt
    21. Clearing the configuration file: ./common/config/nginx/nginx.conf
    22. Clearing the configuration file: ./common/config/jobservice/env
    23. Clearing the configuration file: ./common/config/jobservice/config.yml
    24. loaded secret from file: /data/secretkey
    25. Generated configuration file: ./common/config/nginx/nginx.conf
    26. Generated configuration file: ./common/config/adminserver/env
    27. Generated configuration file: ./common/config/core/env
    28. Generated configuration file: ./common/config/registry/config.yml
    29. Generated configuration file: ./common/config/db/env
    30. Generated configuration file: ./common/config/jobservice/env
    31. Generated configuration file: ./common/config/jobservice/config.yml
    32. Generated configuration file: ./common/config/log/logrotate.conf
    33. Generated configuration file: ./common/config/registryctl/env
    34. Generated configuration file: ./common/config/core/app.conf
    35. Generated certificate, key file: ./common/config/core/private_key.pem, cert file: ./common/config/registry/root.crt
    36. The configuration files are ready, please use docker-compose to start the service.
    37.  
    38.  
    39. [Step 2]: checking existing instance of Harbor ...
    40.  
    41. Note: stopping existing Harbor instance ...
    42. Stopping nginx              ... done
    43. Stopping harbor-jobservice  ... done
    44. Stopping harbor-portal      ... done
    45. Stopping harbor-core        ... done
    46. Stopping redis              ... done
    47. Stopping registryctl        ... done
    48. Stopping registry           ... done
    49. Stopping harbor-db          ... done
    50. Stopping harbor-adminserver ... done
    51. Stopping harbor-log         ... done
    52. Removing nginx              ... done
    53. Removing harbor-jobservice  ... done
    54. Removing harbor-portal      ... done
    55. Removing harbor-core        ... done
    56. Removing redis              ... done
    57. Removing registryctl        ... done
    58. Creating harbor-log ... done
    59. Removing harbor-db          ... done
    60. Removing harbor-adminserver ... done
    61. Removing harbor-log         ... done
    62. Removing network harbor_harbor
    63. Creating harbor-db ... done
    64. Creating harbor-core ... done
    65. [Step 3]: starting Harbor ...
    66. Creating harbor-portal ... done
    67. Creating nginx ... done
    68. Creating redis ... 
    69. Creating harbor-adminserver ... 
    70. Creating registry ... 
    71. Creating harbor-db ... 
    72. Creating registryctl ... 
    73. Creating harbor-core ... 
    74. Creating harbor-portal ... 
    75. Creating harbor-jobservice ... 
    76. Creating nginx ... 
    77.  
    78. ✔ ----Harbor has been installed and started successfully.----
    79.  
    80. Now you should be able to visit the admin portal at http://harbor.jxwrd.gov.cn. 
    81. For more details, please visit https://github.com/goharbor/harbor .
    82.

    再次查看harbor服务状态：

    1. root@harbor:/home/service/harbor# docker-compose ps
    2.        Name                     Command               State                                Ports                              
    3. ------------------------------------------------------------------------------------------------------------------------------
    4. harbor-adminserver   /harbor/start.sh                 Up                                                                      
    5. harbor-core          /harbor/start.sh                 Up                                                                      
    6. harbor-db            /entrypoint.sh postgres          Up      5432/tcp                                                        
    7. harbor-jobservice    /harbor/start.sh                 Up                                                                      
    8. harbor-log           /bin/sh -c /usr/local/bin/ ...   Up      127.0.0.1:1514->10514/tcp                                       
    9. harbor-portal        nginx -g daemon off;             Up      80/tcp                                                          
    10. nginx                nginx -g daemon off;             Up      0.0.0.0:443->443/tcp, 0.0.0.0:4443->4443/tcp, 0.0.0.0:80->80/tcp
    11. redis                docker-entrypoint.sh redis ...   Up      6379/tcp                                                        
    12. registry             /entrypoint.sh /etc/regist ...   Up      5000/tcp                                                        
    13. registryctl          /harbor/start.sh                 Up

    发现harbor服务恢复了，通过访问地址去访问是可以正常打开，并且正常获取到镜像文件的。

    三、总结：

     到此，该harbor访问异常的问题就处理好了，希望问题分析排查的过程对大家有帮助！

  • 相关阅读:
    关于什么是框架
    2、乐趣国学——“君子慎独”
    Mysql笔记
    WPF调用webapi并展示数据(一)：WPF页面的构建
    C语言力扣第50题之Pow（x，n），求x的n次幂。递归算法
    java笔记37，Lambda表达式
    oCPC实践录 | oCPC转化的设计、选择、归因与成本设置（4）
    USB device ‘FTDI Dual RS232-HS‘ with UUID
    java-net-php-python-ssm高校综合素质测评系统计算机毕业设计程序
    基于Hadoop的电商用户分析系统
  • 原文地址：https://blog.csdn.net/jiang0615csdn/article/details/136152553