Flink 集群部署

Flink 集群部署

Flink 集群部署分类

根据以下两种条件将集群部署模式分为三种类型：

1、集群的生命周期和资源隔离；

2、根据程序main()方法执行在Client 还是JobManager

• Session Mode
  • 共享JobManager 和TaskManager，所有提交的Job都在一个Runtime中运行
• Pre-Job Mode
  • 独享JobManager 和TaskManager，好比为每个Job单独启动一个Runtime
• Application Mode （1.11版本提出）
  • Application的main()运行在Cluster上，而不在客户端
  • 每个Application对应一个Runtime，Application中可以包含有多个Job

Flink支持资源管理器

• Standalone
• Hadoop Yarn
• Apache Mesos
• Docker
• Kubernetes

Standalone

单机

https://flink.apache.org/downloads.html

wget https://dlcdn.apache.org/flink/flink-1.16.0/flink-1.16.0-bin-scala_2.12.tgz
tar -zxvf flink-1.16.0-bin-scala_2.12.tgz
cd flink-1.16.0 & ./bin/start-cluster.sh
1
2
3

多机

修改 conf/flink-conf.yaml:

high-availability: zookeeper
high-availability.zookeeper.quorum: node01:2181,node02:2181,node03:2181,node04:2181,node05:2181 
high-availability.zookeeper.path.root: /flink 
#high-availability.cluster-id: /cluster_one # important: customize per cluster 
high-availability.storageDir: hdfs:///flink/recovery
1
2
3
4
5

配置 conf/masters：

localhost:8081 
localhost:8082
1
2

配置 conf/zoo.cfg（可选）：

server.0=localhost:2888:3888
1

启动 HA 集群

$ bin/start-cluster.sh 
Starting HA cluster with 2 masters and 1 peers in ZooKeeper quorum. 
Starting standalonesession daemon on host localhost. 
Starting standalonesession daemon on host localhost. 
Starting taskexecutor daemon on host localhost.
1
2
3
4
5

Yarn

Session 集群启动：

./bin/yarn-session.sh -jm 1024m -tm 4096m
1

Job 集群启动：

./bin/flink run -m yarn-cluster -p 4 -yjm 1024m -ytm 4096m ./examples/batch/WordCout.jar
1

Application Mode集群启动：

./bin/flink run-application -t yarn-application \
-Djobmanager.memory.process.size=2048m \
-Dtaskmanager.memory.process.size=4096m \
-Dyarn.provided.lib.dirs="hdfs://node02:8020/flink-training/flink-1.16.0" \
./MyApplication.jar
1
2
3
4
5