1. JVM设置不合理。
bootstrap check failure [1] of [1]: initial heap size [67108864] not equal to maximum heap size [3221225472]; this can cause resize pauses
原因分析：设置的jvm.options里heap参数不合理，超过操作系统可分配的，需要进行适当的调整。
## -Xms4g
## -Xmx4g
2. network.host配置的域名IP不能对应。
[2022-09-05T01:30:14,364][INFO ][o.e.x.s.c.f.PersistentCache] [es-goya01] persistent cache index loaded
[2022-09-05T01:30:14,365][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [es-goya01] deprecation component started
[2022-09-05T01:30:15,200][ERROR][o.e.b.Elasticsearch ] [es-goya01] fatal exception while booting Elasticsearchorg.elasticsearch.transport.BindTransportException: Failed to bind to [fe80::44da:d53d:e95a:6be6]:[9300-9399]
at org.elasticsearch.server@8.4.0/org.elasticsearch.transport.TcpTransport.bindToPort(TcpTransport.java:486)
at org.elasticsearch.server@8.4.0/org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:447)
at org.elasticsearch.transport.netty4@8.4.0/org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:142)
at org.elasticsearch.security@8.4.0/org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport.doStart(SecurityNetty4Transport.java:96)
at org.elasticsearch.security@8.4.0/org.elasticsearch.xpack.security.transport.netty4.SecurityNetty4ServerTransport.doStart(SecurityNetty4ServerTransport.java:59)
See logs for more details.
ERROR: Elasticsearch did not exit normally - check the logs at /app/elasticsearch/logs/es840-cluster.log
#这里如果写成域名就需要/etc/hosts里添加ip域名信息，或者直接写成IP地址。否则启动elasticsearch会报“Failed to bind to xxx”错误。
3.ClusterFormationFailureHelper，如果报这个错误，需要将三个节点都启动。
[2022-09-05T02:01:03,396][INFO ][o.e.b.BootstrapChecks ] [es-goya01] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2022-09-05T02:01:03,398][INFO ][o.e.c.c.ClusterBootstrapService] [es-goya01] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to [es-goya01, es-goya02, es-goya03]
[2022-09-05T02:01:13,407][WARN ][o.e.c.c.ClusterFormationFailureHelper] [es-goya01] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [es-goya01, es-goya02, es-goya03] to bootstrap a cluster: have discovered [{es-goya01}{wbTopYh5QnGFRWBbJSKulA}{xgqHLLkYS6qR1yGgy1sbRA}{es-goya01}{goya1}{192.168.88.5:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{es-goya01}{wbTopYh5QnGFRWBbJSKulA}{xgqHLLkYS6qR1yGgy1sbRA}{es-goya01}{goya1}{192.168.88.5:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-09-05T02:01:23,409][WARN ][o.e.c.c.ClusterFormationFailureHelper] [es-goya01] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [es-goya01, es-goya02, es-goya03] to bootstrap a cluster: have discovered [{es-goya01}{wbTopYh5QnGFRWBbJSKulA}{xgqHLLkYS6qR1yGgy1sbRA}{es-goya01}{goya1}{192.168.88.5:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{es-goya01}{wbTopYh5QnGFRWBbJSKulA}{xgqHLLkYS6qR1yGgy1sbRA}{es-goya01}{goya1}{192.168.88.5:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-09-05T02:01:33,410][WARN ][o.e.c.c.ClusterFormationFailureHelper] [es-goya01] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [es-goya01, es-goya02, es-goya03] to bootstrap a cluster: have discovered [{es-goya01}{wbTopYh5QnGFRWBbJSKulA}{xgqHLLkYS6qR1yGgy1sbRA}{es-goya01}{goya1}{192.168.88.5:9300}{cdfhilmrstw}]; discovery will continue using [] from hosts providers and [{es-goya01}{wbTopYh5QnGFRWBbJSKulA}{xgqHLLkYS6qR1yGgy1sbRA}{es-goya01}{goya1}{192.168.88.5:9300}{cdfhilmrstw}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2022-09-05T02:01:33,413][WARN ][o.e.n.Node ] [es-goya01] timed out while waiting for initial discovery state - timeout: 30s
4，执行修改密码报错。
[es@goya1 ~]$ elasticsearch-setup-passwords interactive
warning: ignoring JAVA_HOME=/app/elasticsearch/jdk; using bundled JDK
02:56:02.177 [main] WARN org.elasticsearch.common.ssl.DiagnosticTrustManager - failed to establish trust with server at [10.0.2.15]; the server provided a certificate with subject name [CN=goya1], fingerprint [ce1d4470307a5d61b306e697ea8d12fb43544e42], no keyUsage and extendedKeyUsage [serverAuth]; the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate has subject alternative names [IP:192.168.88.5,IP:192.168.88.9,IP:192.168.88.7,DNS:goya3,DNS:goya1,DNS:goya2]; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [e503be851aedc259a13abc3702831b8092486ce2] {trusted issuer}) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is trusted in this ssl context ([xpack.security.http.ssl (with trust configuration: StoreTrustConfig{path=/app/elasticsearch/config/certs/http.p12, password=
java.security.cert.CertificateException: No subject alternative names matching IP address 10.0.2.15 found
at sun.security.util.HostnameChecker.matchIP(HostnameChecker.java:165) ~[?:?]
at sun.security.util.HostnameChecker.match(HostnameChecker.java:101) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:452) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:426) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:238) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:82) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1335) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1226) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1169) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:458) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:201) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1505) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1420) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:578) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:183) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:142) ~[?:?]
at org.elasticsearch.xpack.core.common.socket.SocketAccess.lambda$doPrivileged$0(SocketAccess.java:42) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:569) ~[?:?]
at org.elasticsearch.xpack.core.common.socket.SocketAccess.doPrivileged(SocketAccess.java:41) ~[?:?]
at org.elasticsearch.xpack.core.security.CommandLineHttpClient.execute(CommandLineHttpClient.java:178) ~[?:?]
at org.elasticsearch.xpack.core.security.CommandLineHttpClient.execute(CommandLineHttpClient.java:112) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$SetupCommand.checkElasticKeystorePasswordValid(SetupPasswordTool.java:340) ~[?:?]
at org.elasticsearch.xpack.security.authc.esnative.tool.SetupPasswordTool$InteractiveSetup.execute(SetupPasswordTool.java:203) ~[?:?]
at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54) ~[elasticsearch-8.4.0.jar:8.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85) ~[elasticsearch-cli-8.4.0.jar:8.4.0]
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:94) ~[elasticsearch-cli-8.4.0.jar:8.4.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85) ~[elasticsearch-cli-8.4.0.jar:8.4.0]
at org.elasticsearch.cli.Command.main(Command.java:50) ~[elasticsearch-cli-8.4.0.jar:8.4.0]
at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64) ~[cli-launcher-8.4.0.jar:8.4.0]
SSL connection to https://10.0.2.15:9200/_security/_authenticate?pretty failed: No subject alternative names matching IP address 10.0.2.15 found
Please check the elasticsearch SSL settings under xpack.security.http.ssl.
ERROR: Failed to establish SSL connection to elasticsearch at https://10.0.2.15:9200/_security/_authenticate?pretty.
原因分析：
是因为elasticsearch.yml配置文件的 http.host: [_local_,_site_]，这两个参数如果这样设置，监听的ip地址会很多，参考https://www.elastic.co/guide/en/elasticsearch/reference/8.4/modules-network.html#network-interface-values
例如查看：
[es@goya1 ~]$ netstat -anp |grep 9200
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp6 0 0 10.0.2.15:9200 :::* LISTEN 1744/java
tcp6 0 0 192.168.88.5:9200 :::* LISTEN 1744/java
tcp6 0 0 192.168.2.105:9200 :::* LISTEN 1744/java
tcp6 0 0 127.0.0.1:9200 :::* LISTEN 1744/java
tcp6 0 0 ::1:9200 :::* LISTEN 1744/java
因此，需要将http.host设置未具体的ip地址，例如 http.host: 192.168.88.5
5, 启动es证书相关的报错。
[2022-09-06T17:50:26,301][INFO ][o.e.p.PluginsService ] [es-goya01] no plugins loaded
[2022-09-06T17:50:30,991][INFO ][o.e.e.NodeEnvironment ] [es-goya01] using [1] data paths, mounts [[/ (/dev/mapper/rhel-root)]], net usable_space [41.7gb], net total_space [49.9gb], types [xfs]
[2022-09-06T17:50:30,992][INFO ][o.e.e.NodeEnvironment ] [es-goya01] heap size [1.9gb], compressed ordinary object pointers [true]
[2022-09-06T17:50:31,081][INFO ][o.e.n.Node ] [es-goya01] node name [es-goya01], node ID [8oYrHS05SEm-53MCZrkPlw], cluster name [es840-cluster], roles [data, data_cold, ingest, data_frozen, ml, data_hot, transform, data_content, data_warm, master, remote_cluster_client]
[2022-09-06T17:50:35,012][ERROR][o.e.b.Elasticsearch ] [es-goya01] fatal exception while booting Elasticsearchorg.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.transport.ssl] - cannot read configured [PKCS12] keystore [/app/elasticsearch/config/certs/elastic-certificates.p12] - this is usually caused by an incorrect password
at org.elasticsearch.xcore@8.4.0/org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:605)
at java.base/java.util.HashMap.forEach(HashMap.java:1421)
at java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)
at org.elasticsearch.xcore@8.4.0/org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:601)
at org.elasticsearch.xcore@8.4.0/org.elasticsearch.xpack.core.ssl.SSLService.
See logs for more details.
ERROR: Elasticsearch did not exit normally - check the logs at /app/elasticsearch/logs/es840-cluster.log
......
[2022-09-06T16:07:02,940][INFO ][o.e.e.NodeEnvironment ] [es-goya03] using [1] data paths, mounts [[/ (/dev/mapper/rhel-root)]], net usable_space [44.4gb], net total_space [49.9gb], types [xfs]
[2022-09-06T16:07:02,940][INFO ][o.e.e.NodeEnvironment ] [es-goya03] heap size [1.9gb], compressed ordinary object pointers [true]
[2022-09-06T16:07:03,016][INFO ][o.e.n.Node ] [es-goya03] node name [es-goya03], node ID [z39t-4TITZOy3cxve-KGIA], cluster name [es840-cluster], roles [data_hot, transform, data_content, data_warm, master, remote_cluster_client, data, data_cold, ingest, data_frozen, ml]
[2022-09-06T16:07:07,334][ERROR][o.e.b.Elasticsearch ] [es-goya03] fatal exception while booting Elasticsearchorg.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.security.http.ssl] - cannot read configured [PKCS12] keystore [/app/elasticsearch/config/certs/http.p12] - this is usually caused by an incorrect password
at org.elasticsearch.xcore@8.4.0/org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSslConfigurations$11(SSLService.java:605)
at java.base/java.util.HashMap.forEach(HashMap.java:1421)
at java.base/java.util.Collections$UnmodifiableMap.forEach(Collections.java:1553)
at org.elasticsearch.xcore@8.4.0/org.elasticsearch.xpack.core.ssl.SSLService.loadSslConfigurations(SSLService.java:601)
at org.elasticsearch.xcore@8.4.0/org.elasticsearch.xpack.core.ssl.SSLService.
See logs for more details.
执行命令：
elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password
elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password