kafka关于修改副本数和分区的数的案例实战（也可用作leader节点均衡案例）

　　　　　　　　　　　　　　kafka关于修改副本数和分区的数的案例实战（也可用作leader节点均衡案例）

　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　　作者：尹正杰

一.关于topic分区数的修改

1>.创建1分区1个的topic，名称为yinzhengjie-channel

[root@node101 ~]# kafka-topics.sh --zookeeper node102.yinzhengjie.org.cn: --create --replication-factor  -partitions  --topic yinzhengjie-channel

Created topic "yinzhengjie-channel".

[root@node101 ~]#

2>.查看topic的信息

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel

Topic:yinzhengjie-channel    PartitionCount:    ReplicationFactor:    Configs:

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr: 103                              #可以很明显的看出kafka 的分区数和副本数都是1

[root@node101 ~]#

3>.将之前创建的topic修改为3个分区

[root@node101 ~]# kafka-topics.sh --alter --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel --partitions

WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected

Adding partitions succeeded!

[root@node101 ~]#

4>.再次查看topic的分区数

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel

Topic:yinzhengjie-channel    PartitionCount:    ReplicationFactor:    Configs:

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:                                     #这是第一个分区，它的副本数依然是1一个，当前0号分区的副本数存放在103这个节点上。说明你的数据修改成功啦！

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:

[root@node101 ~]#

二.关于topic副本数的修改

1>.编写分配脚本

[root@node101 ~]# cat addReplicas.json

{"topics":

    [{"topic":"yinzhengjie-channel"}],

    "version":

}

[root@node101 ~]#

2>.执行分配计划，用于生成json格式的文件

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn: --topics-to-move-json-file addReplicas.json --broker-list "101,102,103" --generate

Current partition replica assignment            　　　　　　#这是当前的分区情况，你可以结合--describe参数查看当前的分区情况

{"version":,"partitions":[{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]}]}

Proposed partition reassignment configuration     　　　　#这是推荐分区计划

{"version":,"partitions":[{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]}]}

[root@node101 ~]#

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel　　　　　　#查看当前分区的情况

Topic:yinzhengjie-channel    PartitionCount:    ReplicationFactor:    Configs:

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:

[root@node101 ~]#

3>.Proposed partition reassignment configuration 后是根据命令行的指定的brokerlist生成的分区分配计划json格式。将 Proposed partition reassignment configuration的配置copy保存到一个文件中 topic-reassignment.json并对它进行相应的修改

[root@node101 ~]# cat topic-reassignment.json　　　　#注意，我在复制下来之后，对副本数进行了修改，由之前的1个副本升级为2个副本。

{"version":,"partitions":[{"topic":"yinzhengjie-channel","partition":,"replicas":[,]},{"topic":"yinzhengjie-channel","partition":,"replicas":[,]},{"topic":"yinzhengjie-channel","partition":,"replicas":[,]}]}

[root@node101 ~]#

4>.根据上一步生成的分配计划配置json文件topic-reassignment.json，进行topic的重新分配。

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn: --reassignment-json-file topic-reassignment.json --execute

Current partition replica assignment

{"version":,"partitions":[{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]},{"topic":"yinzhengjie-channel","partition":,"replicas":[]}]}

Save this to use as the --reassignment-json-file option during rollback

Successfully started reassignment of partitions.

[root@node101 ~]#

5>.查看分配的进度

[root@node101 ~]# kafka-reassign-partitions.sh --zookeeper node102.yinzhengjie.org.cn: --reassignment-json-file topic-reassignment.json --verify

Status of partition reassignment:

Reassignment of partition [yinzhengjie-channel,] completed successfully　　　　　　#如果这里的参数是：is still in progress，说明正在进行分配，如果看到当前的提示说明分配完成。

Reassignment of partition [yinzhengjie-channel,] completed successfully

Reassignment of partition [yinzhengjie-channel,] completed successfully

[root@node101 ~]#

　　　　　温馨提示，上述的方法不仅仅可以用来修改副本数，还可以用来修改你的leader节点，下图我就是我在生产环境中用来均衡leader节点的实操截图：（是不是上面我提到的2种状态都有呢？）

6>.如果分配完成，我们再次查看

[root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel　　　　　　#查看当前分区的情况，这是还没有重新分配的时候

Topic:yinzhengjie-channel    PartitionCount:    ReplicationFactor:    Configs:

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr: 103　　　　　　　　#这里的副本数只有一个！

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas:     Isr:

[root@node101 ~]#

root@node101 ~]# kafka-topics.sh --describe --zookeeper node102.yinzhengjie.org.cn: --topic yinzhengjie-channel

Topic:yinzhengjie-channel    PartitionCount:    ReplicationFactor:    Configs:

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas: ,    Isr: ,　　　　　　#副本数编程了2个！

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas: ,    Isr: ,

    Topic: yinzhengjie-channel    Partition:     Leader:     Replicas: ,    Isr: ,

[root@node101 ~]#

巴特西

kafka关于修改副本数和分区的数的案例实战（也可用作leader节点均衡案例）

最新文章

热门文章