java.nio.channels.SocketChannel[connection-pending remote=/xx.xx.xx.xx:9866]

目录

背景

问题描述

解决办法

背景

CDH集群在内网中部署，外网客户端需要正常提交任务到内网集群Yarn上，但外网客户端和内网网络不能直接连通，于是通过将内网中的每台主机绑定一个浮动ip，然后开通外网客户端和浮动ip之间的网络来实现上述需求。

问题描述

外网客户端通过连接浮动ip来提交任务到内网集群，任务提交到Yarn之后，集群返回响应内容给客户端，但响应内容中涉及的节点信息均为内网ip，导致客户端无法连接。具体报错如下：

[INFO] 2023-09-20 16:44:50.515  - [taskAppId=TASK-12637-0-7787]:[138] -  -> 2023-09-20 16:44:49,952 INFO  org.apache.hadoop.hdfs.DataStreamer                          [] - Exception in createBlockOutputStream
	org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/172.17.0.8:9866]
		at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:259) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1692) [flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1648) [flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:704) [flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
	2023-09-20 16:44:49,964 WARN  org.apache.hadoop.hdfs.DataStreamer                          [] - Abandoning BP-1309512692-172.17.0.6-1691719706686:blk_1073803089_62280
	2023-09-20 16:44:49,980 WARN  org.apache.hadoop.hdfs.DataStreamer                          [] - Excluding datanode DatanodeInfoWithStorage[172.17.0.8:9866,DS-961a5b2e-c2a1-46a3-bfdd-3910d2570bb3,DISK]
[INFO] 2023-09-20 16:45:50.524  - [taskAppId=TASK-12637-0-7787]:[138] -  -> 2023-09-20 16:45:50,043 INFO  org.apache.hadoop.hdfs.DataStreamer                          [] - Exception in createBlockOutputStream
	org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/172.17.0.6:9866]
		at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:259) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1692) [flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1648) [flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:704) [flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
	2023-09-20 16:45:50,044 WARN  org.apache.hadoop.hdfs.DataStreamer                          [] - Abandoning BP-1309512692-172.17.0.6-1691719706686:blk_1073803091_62282
	2023-09-20 16:45:50,053 WARN  org.apache.hadoop.hdfs.DataStreamer                          [] - Excluding datanode DatanodeInfoWithStorage[172.17.0.6:9866,DS-3a03d2ae-c218-44f6-80b6-253cb6ada508,DISK]
[INFO] 2023-09-20 16:46:50.415  - [taskAppId=TASK-12637-0-7787]:[127] - shell exit status code：1
[ERROR] 2023-09-20 16:46:50.415  - [taskAppId=TASK-12637-0-7787]:[137] - process has failure , exitStatusCode : 1 , ready to kill ...
[INFO] 2023-09-20 16:46:50.534  - [taskAppId=TASK-12637-0-7787]:[138] -  -> 2023-09-20 16:46:50,083 INFO  org.apache.hadoop.hdfs.DataStreamer                          [] - Exception in createBlockOutputStream
	org.apache.hadoop.net.ConnectTimeoutException: 60000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/172.17.0.4:9866]
		at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:534) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.createSocketForPipeline(DataStreamer.java:259) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1692) [flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1648) [flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:704) [flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
	2023-09-20 16:46:50,084 WARN  org.apache.hadoop.hdfs.DataStreamer                          [] - Abandoning BP-1309512692-172.17.0.6-1691719706686:blk_1073803093_62284
	2023-09-20 16:46:50,091 WARN  org.apache.hadoop.hdfs.DataStreamer                          [] - Excluding datanode DatanodeInfoWithStorage[172.17.0.4:9866,DS-5363866a-d143-42f7-85bb-a8236e0bbc41,DISK]
	2023-09-20 16:46:50,105 WARN  org.apache.hadoop.hdfs.DataStreamer                          [] - DataStreamer Exception
	org.apache.hadoop.ipc.RemoteException: File /user/hdfs/.flink/application_1691720545069_0007/chunjun/bin/chunjun-docker.sh could only be written to 0 of the 1 minReplication nodes. There are 3 datanode(s) running and 3 node(s) are excluded in this operation.
		at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:2102)
		at org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.chooseTargetForNewBlock(FSDirWriteFileOp.java:294)
		at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2673)
		at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:872)
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:550)
		at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
		at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
		at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
		at java.security.AccessController.doPrivileged(Native Method)
		at javax.security.auth.Subject.doAs(Subject.java:422)
		at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
		at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
	
		at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1489) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.ipc.Client.call(Client.java:1435) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.ipc.Client.call(Client.java:1345) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at com.sun.proxy.$Proxy30.addBlock(Unknown Source) ~[?:?]
		at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:444) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_211]
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_211]
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_211]
		at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_211]
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:409) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:163) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:155) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:346) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at com.sun.proxy.$Proxy31.addBlock(Unknown Source) ~[?:?]
		at org.apache.hadoop.hdfs.DataStreamer.locateFollowingBlock(DataStreamer.java:1838) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1638) ~[flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
		at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:704) [flink-shaded-hadoop-2-uber-2.8.3-10.0.jar:2.8.3-10.0]
	2023-09-20 16:46:50,112 ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli                [] - Error while running the Flink session.

解决办法

• 思路1

客户端配置主机映射，将内网ip映射为浮动ip，经过尝试，该方案不可行。

• 思路2

修改HDFS配置

  <property>
     <name>dfs.clientuse.datanode.hostnamename>
     <value>truevalue>
  property>