kafka增加磁盘或者分区，topic重分区

场景：kafka配置文件log.dirs增加了几个目录，但是新目录没有分区数据写入，所以打算进行重分区一下。

1.生成迁移计划

进入kafka/bin目录

新建 topic-reassign.json,把要重分区的topic按下面格式写。

{
          "topics": [{
                       "topic": "VOLTE_GM"
                    },{
                       "topic": "VOLTE_GM_MEDIA"
                    },{
                       "topic": "VOLTE_MW"
                    },{
                       "topic": "VOLTE_RX"
                    },{
                       "topic": "VOLTE_SH"
                    },{
                       "topic": "VOLTE_SV"
                    }],
          "version": 1
}

 执行命令

./kafka-reassign-partitions.sh --zookeeper kafka001:2181/kafka --generate --topics-to-move-json-file topic-reassign.json --broker-list 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39

broker-list 是kafka配置文件中的 broker.id

输出：

Current partition replica assignment：当前分区策略，建议保存，回退的时候会用到。

Proposed partition reassignment configuration：建议分区策略

Current partition replica assignment
{"version":1,"partitions":[{"topic":"VOLTE_GM","partition":1,"replicas":[35,18],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":2,"replicas":[0,19],"log_dirs":["any","any"]},{"topic":"VOLTE_RX","partition":0,"replicas":[33,2],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":1,"replicas":[10,2],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":0,"replicas":[14,13],"log_dirs":["any","any"]},{"topic":"VOLTE_GM","partition":2,"replicas":[36,19],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":2,"replicas":[11,3],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":1,"replicas":[39,18],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":2,"replicas":[16,15],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":1,"replicas":[15,14],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":1,"replicas":[38,23],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":2,"replicas":[39,24],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":0,"replicas":[38,17],"log_dirs":["any","any"]},{"topic":"VOLTE_RX","partition":2,"replicas":[35,4],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":0,"replicas":[37,22],"log_dirs":["any","any"]},{"topic":"VOLTE_GM","partition":0,"replicas":[34,17],"log_dirs":["any","any"]},{"topic":"VOLTE_RX","partition":1,"replicas":[34,3],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":0,"replicas":[9,1],"log_dirs":["any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"VOLTE_RX","partition":0,"replicas":[28,21],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":0,"replicas":[1,25],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":0,"replicas":[36,19],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":2,"replicas":[3,27],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":0,"replicas":[15,13],"log_dirs":["any","any"]},{"topic":"VOLTE_RX","partition":2,"replicas":[30,23],"log_dirs":["any","any"]},{"topic":"VOLTE_GM","partition":1,"replicas":[12,32],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":0,"replicas":[35,25],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":2,"replicas":[17,15],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":2,"replicas":[38,21],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":2,"replicas":[37,27],"log_dirs":["any","any"]},{"topic":"VOLTE_SH","partition":1,"replicas":[2,26],"log_dirs":["any","any"]},{"topic":"VOLTE_RX","partition":1,"replicas":[29,22],"log_dirs":["any","any"]},{"topic":"VOLTE_GM","partition":0,"replicas":[11,31],"log_dirs":["any","any"]},{"topic":"VOLTE_MW","partition":1,"replicas":[16,14],"log_dirs":["any","any"]},{"topic":"VOLTE_GM_MEDIA","partition":1,"replicas":[37,20],"log_dirs":["any","any"]},{"topic":"VOLTE_SV","partition":1,"replicas":[36,26],"log_dirs":["any","any"]},{"topic":"VOLTE_GM","partition":2,"replicas":[13,33],"log_dirs":["any","any"]}]}

 将标红的json保存到topic-todo.json

2.执行迁移命令

./kafka-reassign-partitions.sh --zookeeper kafka001:2181/kafka --reassignment-json-file topic-todo.json --execute --throttle 1048576000

 throttle带宽，单位是B，上面表示100M/s带宽。

3.验证迁移结果

./kafka-reassign-partitions.sh --zookeeper kafka001:2181/kafka --reassignment-json-file topic-todo.json --verify

 

 分区多，数据量大的话，用时多一些，半个小时一个小时也是可能的。