这里是weihubeats,觉得文章不错可以关注公众号小奏技术,文章首发。拒绝营销号,拒绝标题党
继上次分析RocketMQ
线上各种system busy
比较合理能提升性能和缓解system busy
的方法就是修改broker的配置为transientStorePoolEnable = true
今天我们就来实战下
RocketMQ
实现了内存级别的读写分离。
想要开启很简单,我们在broker的配置文件中添加如下配置
transientStorePoolEnable = true
默认是关闭的
开启后需要注意transientStorePoolSize
的大小,默认是5,mappedFileSizeCommitLog
大小默认是1G
private int mappedFileSizeCommitLog = 1024 * 1024 * 1024;
所以如果我们要开启transientStorePoolEnable
,然后其他的不配置的话,我们至少需要预留5g的堆外内存给RocketMQ
使用
比如我们机器是16G,那么JVM的大小最多能给的是16- 5 = 11 G,实际netty
也是使用堆外内存传输的,所以实际最多给到的JVM内存大小可能是8-9G
开启 TransientStorePool 后,消息写入时将写入ByteBuffer.allocateDirect
申请的堆外内存中,由异步刷盘线程写入 fileChannel
中(Page Cache),最后进行进行刷盘。消息读取时,只会从 Page Cache
读取
所以如果消息未刷盘到Page Cache
,broker宕机了,消息就会丢失。
优点:
Page Cache
更快,读写分离,降低锁的占用。缺点:
Page Cache
消息会丢失。其次是消耗更多的内存本次我们将使用RocketMQ
的基准测试脚本分别测试开启读写分离和不开启读写分离的性能。
jvm参数
-server -Xms8g -Xmx8g -XX:MetaspaceSize=128m -XX:MaxMetaspaceSize=320m -XX:+UseG1GC -XX:G1HeapRegionSize=16m -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -XX:SoftRefLRUPolicyMSPerMB=0 -verbose:gc -Xloggc:/dev/shm/rmq_srv_gc_%p_%t.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=30m -XX:-OmitStackTraceInFastThrow -XX:-UseLargePages
4核16G
主要是官方提供的基准测试脚本
JAVA_HOME=/usr sh producer.sh -n 192.168.1.1:9876 -t xiazou-topic -s 3072
JAVA_HOME=/usr sh consumer.sh -n 192.168.1.1:9876 -t xiazou-topic -g gid_xiaozou
压测方式,一个机器同时执行两个脚本,进行生产和消费
机器内存使用在8G
左右
benchmark git:(master) topic: xiazou-topic, threadCount: 64, messageSize: 3072, keyEnable: false, propertySize: 0, tagCount: 0, traceEnable: false, aclEnable: false, messageQuantity: 0, delayEnable: false, delayLevel: 1, asyncEnable: false
Current Time: 2023-10-17 11:59:31,297 | Send TPS: 35634 | Max RT(ms): 421 | Average RT(ms): 1.796 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 11:59:41,297 | Send TPS: 39868 | Max RT(ms): 421 | Average RT(ms): 1.605 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 11:59:51,297 | Send TPS: 37434 | Max RT(ms): 421 | Average RT(ms): 1.709 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 12:00:01,297 | Send TPS: 26596 | Max RT(ms): 421 | Average RT(ms): 2.407 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 12:00:11,297 | Send TPS: 35836 | Max RT(ms): 421 | Average RT(ms): 1.786 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 12:00:21,297 | Send TPS: 36223 | Max RT(ms): 421 | Average RT(ms): 1.767 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 12:00:31,297 | Send TPS: 35910 | Max RT(ms): 421 | Average RT(ms): 1.782 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 12:00:41,297 | Send TPS: 34397 | Max RT(ms): 421 | Average RT(ms): 1.860 | Send Failed: 0 | Response Failed: 0
➜ benchmark git:(master) topic: xiazou-topic, threadCount 20, group: gid_xiaozou, suffix: false, filterType: null, expression: null, msgTraceEnable: false, aclEnable: false
Consumer Started.
Current Time: 2023-10-17 11:59:31,297 | Consume TPS: 35822 | AVG(B2C) RT(ms): 2.776 | AVG(S2C) RT(ms): 2.536 | MAX(B2C) RT(ms): 25 | MAX(S2C) RT(ms): 25 | Consume Fail: 0
Current Time: 2023-10-17 11:59:41,297 | Consume TPS: 36315 | AVG(B2C) RT(ms): 2.741 | AVG(S2C) RT(ms): 2.505 | MAX(B2C) RT(ms): 32 | MAX(S2C) RT(ms): 32 | Consume Fail: 0
Current Time: 2023-10-17 11:59:51,297 | Consume TPS: 36315 | AVG(B2C) RT(ms): 2.680 | AVG(S2C) RT(ms): 2.449 | MAX(B2C) RT(ms): 43 | MAX(S2C) RT(ms): 30 | Consume Fail: 0
生产和消费消息QPS在3.5w左右
机器内存使用在12G
左右
benchmark git:(master) topic: xiazou-topic, threadCount: 64, messageSize: 3072, keyEnable: false, propertySize: 0, tagCount: 0, traceEnable: false, aclEnable: false, messageQuantity: 0, delayEnable: false, delayLevel: 1, asyncEnable: false
Current Time: 2023-10-17 13:59:31,297 | Send TPS: 27799 | Max RT(ms): 163 | Average RT(ms): 2.302 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 13:59:41,297 | Send TPS: 34444 | Max RT(ms): 163 | Average RT(ms): 1.858 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 13:59:51,297 | Send TPS: 37879 | Max RT(ms): 163 | Average RT(ms): 1.689 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:01,297 | Send TPS: 37457 | Max RT(ms): 163 | Average RT(ms): 1.709 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:11,297 | Send TPS: 35222 | Max RT(ms): 163 | Average RT(ms): 1.818 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:21,297 | Send TPS: 41495 | Max RT(ms): 163 | Average RT(ms): 1.542 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:31,297 | Send TPS: 41249 | Max RT(ms): 163 | Average RT(ms): 1.550 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:41,297 | Send TPS: 41962 | Max RT(ms): 163 | Average RT(ms): 1.525 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:00:51,297 | Send TPS: 39707 | Max RT(ms): 523 | Average RT(ms): 1.612 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:01:01,297 | Send TPS: 43444 | Max RT(ms): 523 | Average RT(ms): 1.473 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:01:11,297 | Send TPS: 42391 | Max RT(ms): 523 | Average RT(ms): 1.510 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:01:21,297 | Send TPS: 41915 | Max RT(ms): 523 | Average RT(ms): 1.527 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:01:31,297 | Send TPS: 42221 | Max RT(ms): 523 | Average RT(ms): 1.516 | Send Failed: 0 | Response Failed: 0
Current Time: 2023-10-17 14:01:41,297 | Send TPS: 42480 | Max RT(ms): 523 | Average RT(ms): 1.506 | Send Failed: 0 | Response Failed: 0
➜ benchmark git:(master) topic: xiazou-topic, threadCount 20, group: gid_xiaozou, suffix: false, filterType: null, expression: null, msgTraceEnable: false, aclEnable: false
Consumer Started.
Current Time: 2023-10-17 13:59:36,467 | Consume TPS: 45646 | AVG(B2C) RT(ms): 2135.170 | AVG(S2C) RT(ms): 2135.517 | MAX(B2C) RT(ms): 6164 | MAX(S2C) RT(ms): 6160 | Consume Fail: 0
Current Time: 2023-10-17 13:59:46,467 | Consume TPS: 38940 | AVG(B2C) RT(ms): 60.442 | AVG(S2C) RT(ms): 61.163 | MAX(B2C) RT(ms): 239 | MAX(S2C) RT(ms): 240 | Consume Fail: 0
Current Time: 2023-10-17 13:59:56,467 | Consume TPS: 38094 | AVG(B2C) RT(ms): 39.721 | AVG(S2C) RT(ms): 40.450 | MAX(B2C) RT(ms): 239 | MAX(S2C) RT(ms): 239 | Consume Fail: 0
Current Time: 2023-10-17 14:00:06,467 | Consume TPS: 33780 | AVG(B2C) RT(ms): 52.946 | AVG(S2C) RT(ms): 53.575 | MAX(B2C) RT(ms): 285 | MAX(S2C) RT(ms): 286 | Consume Fail: 0
Current Time: 2023-10-17 14:00:16,467 | Consume TPS: 40338 | AVG(B2C) RT(ms): 48.543 | AVG(S2C) RT(ms): 49.286 | MAX(B2C) RT(ms): 248 | MAX(S2C) RT(ms): 248 | Consume Fail: 0
Current Time: 2023-10-17 14:00:26,467 | Consume TPS: 41998 | AVG(B2C) RT(ms): 66.248 | AVG(S2C) RT(ms): 67.008 | MAX(B2C) RT(ms): 238 | MAX(S2C) RT(ms): 238 | Consume Fail: 0
Current Time: 2023-10-17 14:00:36,467 | Consume TPS: 41713 | AVG(B2C) RT(ms): 72.638 | AVG(S2C) RT(ms): 73.397 | MAX(B2C) RT(ms): 231 | MAX(S2C) RT(ms): 232 | Consume Fail: 0
Current Time: 2023-10-17 14:00:46,467 | Consume TPS: 39853 | AVG(B2C) RT(ms): 53.383 | AVG(S2C) RT(ms): 54.047 | MAX(B2C) RT(ms): 524 | MAX(S2C) RT(ms): 234 | Consume Fail: 0
Current Time: 2023-10-17 14:00:56,467 | Consume TPS: 42399 | AVG(B2C) RT(ms): 59.808 | AVG(S2C) RT(ms): 60.542 | MAX(B2C) RT(ms): 242 | MAX(S2C) RT(ms): 243 | Consume Fail: 0
Current Time: 2023-10-17 14:01:06,467 | Consume TPS: 43756 | AVG(B2C) RT(ms): 75.232 | AVG(S2C) RT(ms): 75.986 | MAX(B2C) RT(ms): 252 | MAX(S2C) RT(ms): 253 | Consume Fail: 0
Current Time: 2023-10-17 14:01:16,467 | Consume TPS: 42434 | AVG(B2C) RT(ms): 57.688 | AVG(S2C) RT(ms): 58.413 | MAX(B2C) RT(ms): 245 | MAX(S2C) RT(ms): 245 | Consume Fail: 0
Current Time: 2023-10-17 14:01:26,467 | Consume TPS: 42152 | AVG(B2C) RT(ms): 69.523 | AVG(S2C) RT(ms): 70.260 | MAX(B2C) RT(ms): 237 | MAX(S2C) RT(ms): 237 | Consume Fail: 0
Current Time: 2023-10-17 14:01:36,467 | Consume TPS: 42024 | AVG(B2C) RT(ms): 53.667 | AVG(S2C) RT(ms): 54.385 | MAX(B2C) RT(ms): 246 | MAX(S2C) RT(ms): 246 | Consume Fail: 0
Current Time: 2023-10-17 14:01:46,467 | Consume TPS: 41974 | AVG(B2C) RT(ms): 71.306 | AVG(S2C) RT(ms): 72.068 | MAX(B2C) RT(ms): 250 | MAX(S2C) RT(ms): 251 | Consume Fail: 0
生产和消费消息QPS在4w左右,相比不开启读写分离QPS生产和消费提高了5000QPS(大概提升了15%的性能),但是RT变长了(每条平均耗时)
总的来说开启读写分离有明显的QPS性能提升,但是会耗费额外的内存,但是每条消息的平均耗时变高了