压测实操--kafka-consumer压测方案
创始人
2024-12-16 19:04:14
0次
作者：九月
环境信息：
操作系统centos7.9，kafka版本为hdp集群中的2.0版本。
Consumer相关参数
使用Kafka自带的kafka-consumer-perf-test.sh脚本进行压测，该脚本参数为：
thread：测试时的单机线程数；
fetch-size：抓取数据量；
partition:分区数，主要是和线程复合来测试；
replication:副本数；
througout:我们所需要的吞吐量，单位时间内处理消息的数量，可能对我们处理消息的延迟有影响；
consumer测试
fetch-size
fetch-size=524288
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size 524288 --threads 4 fetch-size=1048576
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size 1048576 --threads 4 fetch-size=2097152
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size 2097152 --threads 4 fetch-size=3145728
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size 3145728 --threads 4 fetch-size=4194304
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 10000000 --fetch-size 4194304 --threads 4 fetch-size=5242880
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --messages 1000000 --fetch-size 5242880 --threads 4 partitions
创建不同partitions的topic
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create --zookeeper mutong01:2181,mutong02:2181,mutong03:2181 --topic test_kafka_consumer1 --partitions 1 --replication-factor 1 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create --zookeeper mutong01:2181,mutong02:2181,mutong03:2181 --topic test_kafka_consumer2 --partitions 2 --replication-factor 1 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create --zookeeper mutong01:2181,mutong02:2181,mutong03:2181 --topic test_kafka_consumer3 --partitions 3 --replication-factor 1 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-topics.sh --create --zookeeper mutong01:2181,mutong02:2181,mutong03:2181 --topic test_kafka_consumer4 --partitions 4 --replication-factor 1 topic生产数据
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh --topic test_kafka_consumer1 --num-records 100000000 --record-size 687 --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 batch.size=5000 --throughput 30000 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh --topic test_kafka_consumer2 --num-records 100000000 --record-size 687 --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 batch.size=5000 --throughput 30000 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh --topic test_kafka_consumer3 --num-records 100000000 --record-size 687 --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 batch.size=5000 --throughput 30000 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-producer-perf-test.sh --topic test_kafka_consumer4 --num-records 100000000 --record-size 687 --producer-props bootstrap.servers=172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 batch.size=5000 --throughput 30000 消费不同partitions的topic
消费不同partitions的topic，获取不同partition下的吞吐量
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer1 --messages 10000000 --fetch-size 1048576 --threads 4 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer2 --messages 10000000 --fetch-size 1048576 --threads 4 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer3 --messages 10000000 --fetch-size 1048576 --threads 4 /usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer4 --messages 10000000 --fetch-size 1048576 --threads 4 fetch-threads
fetch-thread=1
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576 --messages 10000000 --threads 1 --num-fetch-threads 1 fetch-thread=4
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576 --messages 10000000 --threads 1 --num-fetch-threads 4 fetch-thread=7
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576 --messages 10000000 --threads 1 --num-fetch-threads 7 fetch-thread=10
/usr/hdp/3.1.5.0-152/kafka/bin/kafka-consumer-perf-test.sh --broker-list 172.16.120.30:6667,172.16.120.36:6667,172.16.120.37:6667 --topic test_kafka_consumer --fetch-size 1048576 --messages 10000000 --threads 1 --num-fetch-threads 10 总结
使用kafka自带的kafka-consumer-perf-test.sh脚本对集群消费情况进行压测，根据业务情况设置要测试的messager条数。不同的变量因素，通过修改其中一个变量条件，固定其余变量值，获取对应的吞吐速率。整理不同条件下的指标结果，形成曲线就可以观察具体的消费趋势。
相关内容
热门资讯
透视智能ai！德州局HHpok...
透视智能ai！德州局HHpoker透视脚本，hh poker辅助有用吗，曝光教程（有挂介绍）1、实时...
透视真的(WPK)竟然真的有挂...
透视真的(WPK)竟然真的有挂(透视)如何判断wpk辅助软件的真假(可靠技巧)；1.如何判断wpk辅...
透视智能ai!aapoker安...
透视智能ai!aapoker安装包怎么使用(透视)可以选牌(果然存在有挂)1、下载好aapoker安...
透视系统！hhpoker是内部...
透视系统！hhpoker是内部控制吗，hhpkoer辅助挂是真的吗，教你教程（有挂脚本）1、玩家可以...
透视黑科技(WPK)真是真的有...
透视黑科技(WPK)真是真的有挂(透视)wpk安卓下载辅助(第三方教程)；1、完成wpk安卓下载辅助...
透视讲解!aapoker辅助怎...
透视讲解!aapoker辅助怎么用(透视)可以开辅助器(都是真的是有挂)；1）aapoker辅助怎么...
透视线上！hhpoker有没有...
透视线上！hhpoker有没有辅助，hhpoker为什么一直输，科技教程（有挂方法）；在进入hhpo...
透视黑科技(WPK)好像有挂(...
透视黑科技(WPK)好像有挂(透视)wpk系统是否存在作弊行为(总结教程)；1、不需要AI权限，帮助...
透视插件!aapoker脚本(...
透视插件!aapoker脚本(透视)辅助器可以用(确实是真的有挂)1、aapoker脚本系统规律教程...
透视私人局！hh poker插...
透视私人局！hh poker插件下载，hhpoker真能买到挂吗，微扑克教程（有挂方法）1、在hhp...