• flink-1.14.4启动报错setPreferCheckpointForRecovery(Z)v


    从flink1.12升级到flink1.14,修改了pom.xml的flink-version,打包的时候发现报错:

    1. // 当有较新的 Savepoint 时,作业也会从 Checkpoint 处恢复
    2. env.getCheckpointConfig().setPreferCheckpointForRecovery(true);

     于是屏蔽了这段配置后,打了包,放到yarn上启动,结果出现了下面的报错:

    1. java.lang.NoSuchMethodError: org.apache.flink.streaming.api.environment.CheckpointConfig.setPreferCheckpointForRecovery(Z)V
    2. at com.sitech.csd.ulmp_v2.flink.util.ExecutionEnvUtil.prepare(ExecutionEnvUtil.java:65)
    3. at com.sitech.csd.ulmp_v2.flink.application.BaseApplication.init(BaseApplication.java:93)
    4. at com.sitech.csd.ulmp_v2.flink.application.BaseApplication.init(BaseApplication.java:32)
    5. at com.sitech.csd.ulmp_v2.flink.LogWritingApplication.main(LogWritingApplication.java:19)
    6. at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    7. at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    8. at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    9. at java.lang.reflect.Method.invoke(Method.java:498)
    10. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
    11. at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
    12. at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
    13. at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
    14. at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
    15. at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
    16. at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
    17. at java.security.AccessController.doPrivileged(Native Method)
    18. at javax.security.auth.Subject.doAs(Subject.java:422)
    19. at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
    20. at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
    21. at

    后来发现是scala版本不对,需要的是2.12,但我用了2.11,修改了版本之后,我重新提交,出现如下报错:

    1. 2023-09-13 09:23:33,516 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/ulmp/flink/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
    2. log4j:WARN No appenders could be found for logger (org.apache.hadoop.yarn.ipc.YarnRPC).
    3. log4j:WARN Please initialize the log4j system properly.
    4. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
    5. 2023-09-13 09:23:33,810 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
    6. 2023-09-13 09:23:33,970 INFO org.apache.hadoop.conf.Configuration [] - resource-types.xml not found
    7. 2023-09-13 09:23:33,971 INFO org.apache.hadoop.yarn.util.resource.ResourceUtils [] - Unable to find 'resource-types.xml'.
    8. 2023-09-13 09:23:33,980 WARN org.apache.flink.yarn.YarnClusterDescriptor [] - Neither the HADOOP_CONF_DIR nor the YARN_CONF_DIR environment variable is set. The Flink YARN Client needs one of these to be set to properly load the Hadoop configuration for accessing YARN.
    9. 2023-09-13 09:23:34,014 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - The configured JobManager memory is 768 MB. YARN will allocate 1024 MB to make up an integer multiple of its minimum allocation memory (1024 MB, configured via 'yarn.scheduler.minimum-allocation-mb'). The extra 256 MB may not be used by Flink.
    10. 2023-09-13 09:23:34,015 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=2048, slotsPerTaskManager=1}
    11. 2023-09-13 09:23:34,471 WARN org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory [] - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
    12. 2023-09-13 09:23:36,566 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting application master application_1687851042699_2369
    13. 2023-09-13 09:23:36,598 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted application application_1687851042699_2369
    14. 2023-09-13 09:23:36,598 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for the cluster to be allocated
    15. 2023-09-13 09:23:36,599 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying cluster, current state ACCEPTED
    16. 2023-09-13 09:24:36,761 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
    17. 2023-09-13 09:24:37,012 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
    18. 2023-09-13 09:24:37,262 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
    19. 2023-09-13 09:24:37,513 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster
    20. 2023-09-13 09:24:37,764 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster

  • 相关阅读:
    【csharp】抽象类与接口有哪些不同?什么时候应该使用抽象类?
    Es6 箭头函数
    element-ui在项目当中的引入以及按需引入使用
    了解Ajax(第一天)
    微信公众号订阅通知/一次性订阅通知
    oracle分区表和分区表exchange
    Ansible-Playbook 剧本语法
    版本控制 | 想要成为硬件设计高手?最佳实践了解一下!
    关于序列化协议,你需要知道的一些内容(2)
    Redis_03_Redis发布订阅(Pub/Sub)
  • 原文地址:https://blog.csdn.net/sinat_23030553/article/details/132845371