gitlab 16.x - ERR unknown command ‘HELLO‘

现象

gitlab部分操作报错500。通过Rails日志发现以下报错：

报错：

RedisClient::CommandError

ERR unknown command 'HELLO'

{
    "severity": "ERROR",
    "time": "2024-04-22T02:50:16.906Z",
    "correlation_id": "01HW1VC67EJEGKQ1A1CW9TMVPV",
    "meta.caller_id": "UserSettings::PersonalAccessTokensController#create",
    "meta.remote_ip": "10.1.11.11",
    "meta.feature_category": "system_access",
    "meta.user": "root",
    "meta.user_id": 1,
    "meta.client_id": "user/1",
    "exception.class": "RedisClient::CommandError",
    "exception.message": "ERR unknown command 'HELLO'",
    "exception.backtrace": [
        "lib/gitlab/instrumentation/redis_client_middleware.rb:26:in `block in call_pipelined'",
        "lib/gitlab/instrumentation/redis_helper.rb:17:in `instrument_call'",
        "lib/gitlab/instrumentation/redis_client_middleware.rb:25:in `call_pipelined'",
        "lib/gitlab/patch/redis_client.rb:12:in `ensure_connected'",
        "config/initializers/forbid_sidekiq_in_transactions.rb:82:in `block (2 levels) in \u003cmodule:NoEnqueueingFromTransactions\u003e'",
        "app/services/notification_service.rb:95:in `access_token_created'",
        "app/services/personal_access_tokens/create_service.rb:20:in `execute'",
        "app/controllers/user_settings/personal_access_tokens_controller.rb:33:in `create'",
        "app/controllers/application_controller.rb:468:in `set_current_admin'",
        "lib/gitlab/session.rb:11:in `with_session'",
        "app/controllers/application_controller.rb:459:in `set_session_storage'",
        "lib/gitlab/i18n.rb:114:in `with_locale'",
        "lib/gitlab/i18n.rb:120:in `with_user_locale'",
        "app/controllers/application_controller.rb:450:in `set_locale'",
        "app/controllers/application_controller.rb:443:in `set_current_context'",
        "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
        "lib/gitlab/middleware/memory_report.rb:13:in `call'",
        "lib/gitlab/middleware/speedscope.rb:13:in `call'",
        "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
        "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
        "lib/gitlab/etag_caching/middleware.rb:21:in `call'",
        "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
        "lib/gitlab/metrics/web_transaction.rb:46:in `run'",
        "lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
        "lib/gitlab/middleware/go.rb:20:in `call'",
        "lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
        "lib/gitlab/database/query_analyzer.rb:40:in `within'",
        "lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
        "lib/gitlab/middleware/multipart.rb:173:in `call'",
        "lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
        "lib/gitlab/middleware/read_only.rb:18:in `call'",
        "lib/gitlab/middleware/unauthenticated_session_expiry.rb:18:in `call'",
        "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
        "lib/gitlab/middleware/path_traversal_check.rb:35:in `call'",
        "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
        "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
        "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
        "lib/gitlab/middleware/request_context.rb:15:in `call'",
        "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
        "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
        "lib/gitlab/middleware/compressed_json.rb:44:in `call'",
        "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
        "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
        "lib/gitlab/metrics/requests_rack_middleware.rb:79:in `call'",
        "lib/gitlab/middleware/release_env.rb:13:in `call'"
    ],
    "user.username": "root",
    "tags.program": "web",
    "tags.locale": "en",
    "tags.feature_category": "system_access",
    "tags.correlation_id": "01HW1VC67EJEGKQ1A1CW9TMVPV",
    "extra.storage": "queues"
}

排查

日志报错是Redis执行HELLO命令报错，这个命令适用于指定RESP版本。

1. Gitlab官方文档要求16.0以上版本，需要使用Redis 6.x or 7.x
2. Redis 6.x 开始支持RESP3
3. 连接到Redis，执行Hello 2成功；执行Hello 3报错：ERR unknown command 'HELLO'

结合以上3点，怀疑Gitlab 16.x版本之前使用RESP2，之后使用RESP3。

进入Redis，执行monitor，复现问题查看操作：

再次看到与HELLO 3命令相关。

结论&后续处理

结论（踩坑）

1. 部分公有云Redis 6.x版本，不支持RESP3协议。
2. Gitlab官方虽然明确说明16.x要使用Redis 6.x or 7.x，但是没有阐述原因。我想其中一个原因就是新的ruby代码对于RESP3协议的依赖。如果官网明确说明，并且附上前置检查步骤会更好。

后续处理

1. 可以自己部署6.x版本redis高可用节点。
2. 可以选择支持RESP3协议的公有云Redis实例使用。