flink on yarn

FlinkSQL on Yarn

root@flink-jobmanager-0:/opt## /opt/flink/bin/flink run -m yarn-cluster -ys 4 -ynm V2xRTDataBackupKafka2JDBC -ynm 1024 -ytm 2048  -c com.nufront.bigdata.v2x.job.V2xRTDataBackupKafka2JDBC /opt/v2x-1.0-SNAPSHOT.jar

...
2022-11-06 12:51:34,034 INFO  org.apache.hadoop.hive.conf.HiveConf                         [] - Found configuration file null

------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Configured default database ahivedatabase doesn't exist in catalog my_catalog.
        at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
        

1
2
3
4
5
6
7
8
9
10
11
12

配置环境变量

System.setProperty("HIVE_CONF_DIR", "/opt/flink/conf");
1

直接通过 yarn application -kill xxx ，checkpoint 被删除…

奇怪的问题

Configured default database ahivedatabase doesn’t exist in catalog my_catalog.

Caused by: java.io.IOException: Unable to open BLOB Server in specified port range: 6124

Savepoint

root@flink-jobmanager-0:/opt/flink/bin# ./flink stop a04f8b4257c0094145187d7556e8272f -yid application_1669559512930_0004 -p hdfs://hdfs-namenode-service:9000/flink-savepoints
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/flink/lib/log4j-slf4j-impl-2.16.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/flink/lib/phoenix-5.0.0-HBase-2.0-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/flink/lib/phoenix-core-5.0.0-HBase-2.0-jar-with-dependencies-with-hbase-site.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Suspending job "a04f8b4257c0094145187d7556e8272f" with a savepoint.
log4j:WARN No appenders could be found for logger (org.apache.hadoop.yarn.client.RMProxy).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
2022-12-20 16:26:36,068 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2022-12-20 16:26:36,166 INFO  org.apache.flink.yarn.YarnClusterDescriptor                  [] - Found Web Interface yarn-nodemanager-0.yarn-nodemanager-service.default.svc.cluster.local:35286 of application 'application_1669559512930_0004'.
Savepoint completed. Path: hdfs://hdfs-namenode-service:9000/flink-savepoints/savepoint-a04f8b-93042a571592
1
2
3
4
5
6
7
8
9
10
11
12
13
14