C++实测无锁队列concurrentqueue、boost.spinlock 和 std::mutex 在多线程情况下的性能表现

测试多线程使用ConcurrentQueue、spinlock和mutex的性能的程序

仓库代码：https://gitee.com/liudegui/lock_test

• 测试mutex 和spin lock的区别，参考mutex 和spin lock的区别;
• 多线程读写concurrentqueue
• 单线程读写readerwriterqueue
• 测试验证基于C++ STL利用CAS原子操作封装的无锁list

摘要

实测无锁队列concurrentqueue、boost.spinlock 和 std::mutex 在多线程情况下的表现;

结论：

已经使用std::mutex的std::list或者std::deque，如果Item结构体比较小，使用boost.spinlock代替std::mutex；如果Item比较大，使用std::mutex；新的业务代码可以酌情使用无锁队列concurrentqueue

ConcurrentQueue、spinlock和mutex对比测试结果

1、30线程并发，1万条Item数据push和pop, Item大小2K左右

• 1.230测试结果

ConcurrentQueue pushTime: 595.476
ConcurrentQueue popTime: 328.856
spinlock pushTime: 412.675
spinlock popTime: 955.207
mutex pushTime: 946.301
mutex popTime: 907.553
1
2
3
4
5
6

• 1.240测试结果

ConcurrentQueue pushTime: 1584.1
ConcurrentQueue popTime: 333.36
spinlock pushTime: 576.209
spinlock popTime: 1479.5
mutex pushTime: 1133.68
mutex popTime: 1107.63
1
2
3
4
5
6

• 1.30测试结果

ConcurrentQueue pushTime: 1005.89
ConcurrentQueue popTime: 244.84
spinlock pushTime: 355.606
spinlock popTime: 402.343
mutex pushTime: 597.448
mutex popTime: 739.805
1
2
3
4
5
6

• 本地linux机器测试结果

ConcurrentQueue pushTime: 140.899
ConcurrentQueue popTime: 80.4264
spinlock pushTime: 136.703
spinlock popTime: 136.91
mutex pushTime: 231.019
mutex popTime: 213.732
1
2
3
4
5
6

• windows平台测试结果

ConcurrentQueue pushTime: 200.119
ConcurrentQueue popTime: 142.239
spinlock pushTime: 602.542
spinlock popTime: 482.394
mutex pushTime: 483.498
mutex popTime: 306.393
1
2
3
4
5
6

• 总结：

linux平台

对于push数据：spinlock > ConcurrentQueue > mutex

对于pop数据: ConcurrentQueue > spinlock > mutex

windows平台

各项都是 ConcurrentQueue 最优

2、30线程并发，1万条Item数据push和pop, Item大小20K左右

• 1.230测试结果

ConcurrentQueue pushTime: 6936.41
ConcurrentQueue popTime: 732.457
spinlock pushTime: 4256.77
spinlock popTime: 7103.61
mutex pushTime: 5165.12
mutex popTime: 4044.51
1
2
3
4
5
6

• 1.240测试结果

ConcurrentQueue pushTime: 18411.2
ConcurrentQueue popTime: 942.713
spinlock pushTime: 5750.07
spinlock popTime: 11232.5
mutex pushTime: 7236.02
mutex popTime: 6573.35
1
2
3
4
5
6

• 1.30测试结果

ConcurrentQueue pushTime: 5399.57
ConcurrentQueue popTime: 247.965
spinlock pushTime: 2253.27
spinlock popTime: 1285.47
mutex pushTime: 2625.55
mutex popTime: 1588.46
1
2
3
4
5
6

• 本地linux机器测试结果

ConcurrentQueue pushTime: 2231.09
ConcurrentQueue popTime: 183.022
spinlock pushTime: 1117.93
spinlock popTime: 715.601
mutex pushTime: 1288.95
mutex popTime: 805.378
1
2
3
4
5
6

• windows平台测试结果

ConcurrentQueue pushTime: 8047.59
ConcurrentQueue popTime: 5098.22
spinlock pushTime: 16736.2
spinlock popTime: 26468.2
mutex pushTime: 21498.3
mutex popTime: 50173.6
1
2
3
4
5
6

• 总结：随着Item大小的增大，使用spinlock 不一定更优，性能反而下降，ConcurrentQueue 的插入性能更明显。

linux平台

**对于push数据：spinlock > mutex > ConcurrentQueue **

**对于pop数据: ConcurrentQueue > mutex> spinlock > **

windows平台

各项都是 ConcurrentQueue 最优

硬件配置

本地linux配置

cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
      8  Intel(R) Core(TM) i7-9700 CPU @ 3.00GHz
      
$ cat /proc/meminfo
MemTotal:       16173836 kB (16G)
MemFree:         4756828 kB
MemAvailable:    7295112 kB
$  uname -a
Linux dgliu-t 5.4.0-47-generic #51~18.04.1-Ubuntu SMP Sat Sep 5 14:35:50 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
1
2
3
4
5
6
7
8
9

1.240

$ cat /proc/cpuinfo | grep asimd | cut -f2 -d: | uniq -c
     96  fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp
     
$ cat /proc/meminfo
MemTotal:       263858208 kB (256G)
MemFree:        30753924 kB
MemAvailable:   259320588 kB
$  uname -a
Linux ubuntu 4.15.0-71-generic #2 SMP Wed Jan 1 23:17:17 EST 2020 aarch64 aarch64 aarch64 GNU/Linux
1
2
3
4
5
6
7
8
9

1.230

$ cat /proc/cpuinfo | grep asimd | cut -f2 -d: | uniq -c
     96  fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma dcpop asimddp
$ cat /proc/meminfo
MemTotal:       65670672 kB  (64G)
MemFree:        48536012 kB
MemAvailable:   62547104 k
$  uname -a
Linux linux 4.15.0-71-generic #4 SMP Mon Dec 2 11:08:59 CST 2019 aarch64 aarch64 aarch64 GNU/Linux
1
2
3
4
5
6
7
8

1.30

$ cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c
$ cat /proc/meminfo
MemTotal:       263836736 kB (256G)
MemFree:        83968488 kB
MemAvailable:   202582764 kB
$  uname -a
Linux test-B8026T70AV2E24HR 4.15.0-45-generic #48~16.04.1-Ubuntu SMP Tue Jan 29 18:03:48 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
1
2
3
4
5
6
7

windows 10 x64

CPU
    Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz
内存
    8.0 GB
    速度: 1867 MHz
    可用  1.5 GB
1
2
3
4
5
6

其它

测试验证基于C++ STL利用CAS原子操作封装的无锁list效果不佳，可能原因是新版gcc已做优化。不推荐参考这篇文章的内容。

可参考文章

atomic, spinlock and mutex性能比较

mutex 和spin lock的区别