Hadoop搭建HA遇到的坑

Hadoop搭建HA遇到的坑

重要报错信息

Zookeeper日志

• Cannot open channel to 2 at election address node03/172.17.0.4:3888
  java.net.ConnectException: Connection refused (Connection refused)

Hadoop日志

• 2021-01-30 19:44:04,310 ERROR org.apache.hadoop.ha.NodeFencer: Unable to fence service by any configured method.
• 2021-01-30 19:44:04,310 WARN org.apache.hadoop.ha.ActiveStandbyElector: Exception handling the winning of election
  java.lang.RuntimeException: Unable to fence NameNode at node01/172.17.0.2:8020
• java.io.EOFException: End of File Exception between local host is: “node02/172.17.0.3”; destination host is: “node02”:8020; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException
• Caused by: java.io.EOFException
• 2021-01-30 19:44:06,535 WARN org.apache.hadoop.ha.FailoverController: Unable to gracefully make NameNode at node01/172.17.0.2:8020 standby (unable to connect)
  java.net.ConnectException: Call From node02/172.17.0.3 to node01:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
• 2021-01-30 19:44:06,535 WARN org.apache.hadoop.ha.SshFenceByTcpPort: Unable to create SSH session
  com.jcraft.jsch.JSchException: invalid privatekey: [B@1bfa0880

结构如下

HOST

NN

NN

JNN

DN

ZKFC

ZK

node01

*

*

*

node02

*

*

*

*

*

node03

*

*

*

*

node04

*

*

问题

• 所有的节点角色均可正常运行，当我将node01中的Active状态的NameNode停止，node02中的NameNode并不会由StandBy状态转变为Active，而重新将node01中的NameNode启动后，node02中的NameNode转变为Active状态。

解决过程

1. 尝试过将各个节点Zookeeper中的zoo.cfg各自的地址改为0.0.0.0，失败
2. 时间已经设置同步
3. 各个角色运行正常
4. apt安装psmisc，失败
5. 网络上说ssh-keygen版本过高高的时候生成秘钥会有问题，于是使用ssh-keygen -t rsa -m PEM，失败
6. 最后发现所有的NameNode应当都对其余节点免密，而我只设置了node01，所以将node02的公钥发送给其余的节点后，故障得以解决

原因

• 如上结构图所示，当node02未配置与2，3，4免密时，相当于运行在node02的zkfc不能与Zookeeper集群通信