• Oracle-RAC集群网络，IO性能测试

    前言:

    Oracle RAC集群对于网络，IO性能有一定的要求，特别是心跳网络，需要千兆以上，Oracle官方最佳实践建议配置万兆网络，而IO性能则是直接影响数据库的用户操作执行，因此，在进行RAC集群的部署时，我们需要对集群网络，IO性能进行测试，以确保性能指标满足集群运行要求。

    本文接下来的内容将讲述Oracle RAC集群网络，IO性能常用的测试工具以及测试方法，希望对大家有所帮助。

    测试工具:

    网络:netpref

    IO:orion,DBMS_RESOURCE_MANAGER.CALIBRATE_IO,fio

    网络性能压测步骤(netpref)

    安装编译netpref工具

    1. tar -zxvf netperf-2.7.0.tar.gz
    2. cd netperf-netperf-2.7.0/
    3. ./configure
    4. make
    5. make install

    验证安装成功

    1. [root@rac1 ~]# netperf -h
    2. ​
    3. Usage: netperf [global options] -- [test options] 
    4. ​
    5. Global options:
    6.     -a send,recv      Set the local send,recv buffer alignment
    7.     -A send,recv      Set the remote send,recv buffer alignment
    8.     -B brandstr       Specify a string to be emitted with brief output
    9.     -c [cpu_rate]     Report local CPU usage
    10.     -C [cpu_rate]     Report remote CPU usage
    11.     -d                Increase debugging output
    12.     -D time,[units] * Display interim results at least every time interval
    13.                       using units as the initial guess for units per second
    14.                       A negative value for time will make heavy use of the
    15.                       system's timestamping functionality
    16.     -f G|M|K|g|m|k    Set the output units
    17.     -F lfill[,rfill]* Pre-fill buffers with data from specified file
    18.     -h                Display this text
    19.     -H name|ip,fam *  Specify the target machine and/or local ip and family
    20.     -i max,min        Specify the max and min number of iterations (15,1)
    21.     -I lvl[,intvl]    Specify confidence level (95 or 99) (99) 
    22.                       and confidence interval in percentage (10)
    23.     -j                Keep additional timing statistics
    24.     -l testlen        Specify test duration (>0 secs) (<0 bytes|trans)
    25.     -L name|ip,fam *  Specify the local ip|name and address family
    26.     -o send,recv      Set the local send,recv buffer offsets
    27.     -O send,recv      Set the remote send,recv buffer offset
    28.     -n numcpu         Set the number of processors for CPU util
    29.     -N                Establish no control connection, do 'send' side only
    30.     -p port,lport*    Specify netserver port number and/or local port
    31.     -P 0|1            Don't/Do display test headers
    32.     -r                Allow confidence to be hit on result only
    33.     -s seconds        Wait seconds between test setup and test start
    34.     -S                Set SO_KEEPALIVE on the data connection
    35.     -t testname       Specify test to perform
    36.     -T lcpu,rcpu      Request netperf/netserver be bound to local/remote cpu
    37.     -v verbosity      Specify the verbosity level
    38.     -W send,recv      Set the number of send,recv buffers
    39.     -v level          Set the verbosity level (default 1, min 0)
    40.     -V                Display the netperf version and exit
    41.     -y local,remote   Set the socket priority
    42.     -Y local,remote   Set the IP_TOS. Use hexadecimal.
    43.     -Z passphrase     Set and pass to netserver a passphrase
    44. ​
    45. For those options taking two parms, at least one must be specified;
    46. specifying one value without a comma will set both parms to that
    47. value, specifying a value with a leading comma will set just the second
    48. parm, a value with a trailing comma will set just the first. To set
    49. each parm to unique values, specify both and separate them with a
    50. comma.
    51. ​
    52. * For these options taking two parms, specifying one value with no comma
    53. will only set the first parms and will leave the second at the default
    54. value. To set the second value it must be preceded with a comma or be a
    55. comma-separated pair. This is to retain previous netperf behaviour.

    进行网络性能压测，netpref通过模拟服务端与客户端的TCP/UDP包网络发送，来测试主机之间的网络性能，这里测试的RAC为两节点的集群，所以节点一与节点二将分别作为服务段以及客户端进行测试

    节点一RAC1作为服务端，启动netserver

    1. [root@rac1 tmp]# netserver 
    2. Starting netserver with host 'IN(6)ADDR_ANY' port '12865' and family AF_UNSPEC
    3. #关闭netserver​,可以通过kill方式

    节点二RAC2作为客户端，发送TCP/UDP进行测试

    进行TCP协议测试

    1. 参数说明
    2. #TCP_RR 在一个TCP连接中进行多次TCP request和response的交易过程，这种模式常常出现在数据库应用中
    3. #TCP_STREAM: TCP批量传输，在测试过程中，netperf向netserver发送批量的TCP数据分组，以确定数据传输过程中的吞吐量
    4. #-H 主机名IP
    5. #-l 时长
    6. [root@rac2 netperf-netperf-2.7.0]# netperf -t TCP_RR -H rac1-priv -l 60
    7. MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to rac1-priv () port 0 AF_INET : first burst 0
    8. Local /Remote
    9. Socket Size   Request  Resp.   Elapsed  Trans.
    10. Send   Recv   Size     Size    Time     Rate         
    11. bytes  Bytes  bytes    bytes   secs.    per sec   
    12. ​
    13. 16384  87380  1        1       60.00    1721.92   
    14. 16384  87380 
    15. #Trans Rate per sec 每秒的交易1721/s
    16. [root@rac2 netperf-netperf-2.7.0]# netperf -t TCP_STREAM -H rac1-priv -l 60
    17. MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to rac1-priv () port 0 AF_INET
    18. Recv   Send    Send                          
    19. Socket Socket  Message  Elapsed              
    20. Size   Size    Size     Time     Throughput  
    21. bytes  bytes   bytes    secs.    10^6bits/sec  
    22. ​
    23.  87380  16384  16384    60.00    2808.18   
    24.  
    25.  #Throughput 10^6bits/sec 网络吞吐量每秒2808Mbps/s

    进行UDP协议测试

    1. 参数说明
    2. #UDP_RR 进行多次UDP request和response的交易过程，这种模式常常出现在数据库应用中
    3. #UDP_STREAM: UDP批量传输，在测试过程中，netperf向netserver发送批量的UDP数据分组，以确定数据传输过程中的吞吐量
    4. #-H 主机名IP
    5. #-l 时长
    6. [root@rac2 netperf-netperf-2.7.0]# netperf -t UDP_RR -H rac1-priv  -l 60
    7. MIGRATED UDP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to rac1-priv () port 0 AF_INET : first burst 0
    8. Local /Remote
    9. Socket Size   Request  Resp.   Elapsed  Trans.
    10. Send   Recv   Size     Size    Time     Rate         
    11. bytes  Bytes  bytes    bytes   secs.    per sec   
    12. ​
    13. 262144 262144 1        1       60.00    1727.69   
    14. 262144 262144
    15. #Trans Rate per sec 每秒的交易1727/s
    16. [root@rac2 netperf-netperf-2.7.0]# netperf -t UDP_STREAM -H rac1-priv -l 60
    17. MIGRATED UDP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to rac1-priv () port 0 AF_INET
    18. Socket  Message  Elapsed      Messages                
    19. Size    Size     Time         Okay Errors   Throughput
    20. bytes   bytes    secs            #      #   10^6bits/sec
    21. ​
    22. 262144   65507   60.00       44194      0     386.00
    23. 262144           60.00       44188            385.95
    24. #Throughput 10^6bits/sec 网络吞吐量每秒385Mbps/s

    总结:通过netpref的测试，我们可以看到TCP_RR的指标为每秒1721/s,TCP_STREAM每秒的吞吐量为2808Mbps/s,UDP_RR的指标为每秒1727/s,UDP_STREAM每秒的吞吐量为385Mbps/s,这大概就是千兆网络的性能

    IO性能压测步骤(orion)

    ORION (Oracle I/O Calibration Tool) 是校准用于 Oracle 数据库的存储系统 I/O 性能的独立工具。校准结果对于了解存储系统的性能有很大帮助，不仅可以找出影响 Oracle 数据库性能的问题，还能测量新数据库安装的大小。由于 ORION 是一个Oracle软件自带的独立工具，用户不需要创建和运行 Oracle 数据库

    查看工具orion

    1. [root@rac1 tmp]# /u01/app/oracle/product/11.2.0/db_1/bin/orion -help
    2. ORION: ORacle IO Numbers -- Version 11.2.0.4.0
    3. ​
    4. ORION runs IO performance tests that model Oracle RDBMS IO workloads.
    5. It measures the performance of small (2-32K) IOs and large (128K+) IOs
    6. at various load levels. 
    7. ​
    8. Each Orion data point is a test for a specific mix of small and large 
    9. IO loads sustained for a duration.  An Orion test consists of multiple 
    10. data point tests.  These data point tests can be represented as a 
    11. two-dimensional matrix.  Each column in the matrix represents data 
    12. point tests with the same small IO load, but varying large IO loads.  
    13. Each row represents data point tests with the same large IO load, but 
    14. varying small IO loads.  An Orion test can be for a single point, a 
    15. single row, a single column, or the whole matrix.
    16. ​
    17. The 'run' parameter is the only mandatory parameter. Defaults
    18. are indicated for all other parameters.  For additional information on
    19. the user interface, see the Orion User Guide.
    20. ​
    21. <testname> is the prefix used for all input and output filenames.  By
    22. default, it is 'orion'.  It can be specified with the 'testname' 
    23. parameter.
    24. ​
    25. <testname>.lun should contain a carriage-return-separated list of LUNs.
    26. ​
    27. The output files for a test run are prefixed by <testname>_<date> where
    28. date is "yyyymmdd_hhmm".
    29. ​
    30. The output files are: 
    31. <testname>_<date>_summary.txt -  Summary of the input parameters, along with 
    32.          the minimum small IO latency (in usecs), the maximum 
    33.          MBPS, and the maximum IOPS observed. 
    34. <testname>_<date>_mbps.csv - Performance results of large IOs in MBPS.
    35. <testname>_<date>_iops.csv - Performance results of small IOs in IOPS.
    36. <testname>_<date>_lat.csv - Latency of small IOs in microseconds.
    37. <testname>_<date>_hist.csv - Histogram of IO latencies.
    38. <testname>_<date>_trace.txt - Extended, unprocessed output.
    39. ​
    40. WARNING: IF YOU ARE PERFORMING WRITE TESTS, BE PREPARED TO LOSE ANY DATA STORED
    41. ON THE LUNS.
    42. ​
    43. Mandatory parameters:
    44. run      Type of workload to run (simple, normal, advanced, dss, oltp).
    45.         simple   - Tests random small (8K) IOs at various loads,
    46.                    then random large (1M) IOs at various loads.
    47.         normal   - Tests combinations of random small (8K) IOs and 
    48.                    random large (1M) IOs.
    49.         advanced - Tests the workload specified by the user 
    50.                    using optional parameters.
    51.         dss      - Tests with random large (1M) IOs at increasing 
    52.                    loads to determine the maximum throughput.
    53.         oltp     - Tests with random small (8K) IOs at increasing 
    54.                    loads to determine the maximum IOPS.
    55. ​
    56. Optional parameters:
    57. testname    Name of the test run.
    58. ​
    59. num_disks    Number of disks (physical spindles).  This number is 
    60.       used to gauge the range of loads that Orion should test 
    61.       at.  Increasing this parameter results in Orion using 
    62.       heavier IO loads.  Default is the number of LUNs in 
    63.       <testname>.lun.
    64. ​
    65. size_small    Size of small IOs in KB.  Default is 8.
    66. ​
    67. size_large    Size of large IOs in KB.  Default is 1024.
    68. ​
    69. type      Type of large IOs (rand, seq).  Default is rand.
    70.         rand - Randomly distributed large IOs.
    71.         seq  - Sequential streams of large IOs.
    72. ​
    73. num_streamIO    Number of concurrent IOs per stream.  This parameter is only 
    74.       used if -type is seq.  Default is 4.
    75. ​
    76. simulate    Orion tests on a virtual LUN formed by combining the
    77.       specified LUNs in one of these ways.  This parameter is 
    78.       typically only used if -type is seq.  Default is concat.
    79.         concat - A serial concatenation of the LUNs.  Each 
    80.                  sequential stream issues IOs to only one LUN. 
    81.         raid0  - A RAID-0 mapping across the LUNs.  Each 
    82.                  sequential stream issues IOs across all LUNs, 
    83.                  using RAID-0 striping.
    84. ​
    85. write      Percentage of IOs that are writes (SEE WARNING ABOVE).
    86.       Default is 0.
    87. ​
    88. cache_size    Size in MBs of the array's cache.
    89.       Unless this option is set to 0, Orion issues a number
    90.       of unmeasured, random IOs before each large sequential
    91.       data point.  These IOs fill up the storage array's cache
    92.       (if any) with random data so that IOs from one
    93.       data point do not result in cache hits for the next
    94.       data point.  Read tests are preceded with junk reads
    95.       and write tests are preceded with junk writes.  If
    96.       specified, this 'cache warming' is performed until
    97.       cache_size MBs of IO have been read or written.
    98.       Default behavior is to issue 2 minutes of unmeasured random 
    99.       IOs before each data point.
    100. ​
    101. duration    Duration of each data point in seconds.  Default is 60.
    102. ​
    103. num_small    Number of outstanding small IOs.  This parameter controls the 
    104.       small IO load.  Only used if -matrix is point, col, or max.  
    105.       No default.
    106. ​
    107. num_large    This parameter controls the large IO load.
    108.       For -type rand, number of outstanding large IOs.
    109.       For -type seq, number of sequential IO streams.  Only used 
    110.       if -matrix is point, row, or max.  No default.
    111. ​
    112. matrix      An Orion test consists of multiple data point tests.  These data 
    113.       point tests can be represented as a two-dimensional matrix. 
    114.       Each column in the matrix represents data point tests with 
    115.       the same small IO load, but varying large IO loads.  Each 
    116.       row represents data point tests with the same large IO load, 
    117.       but varying small IO loads.  An Orion test can be for a 
    118.       single point, a single row, a single column, or the whole 
    119.       matrix, depending on the matrix option setting below. 
    120.       Default is basic.
    121.         basic    - Test small IOs only, then large IOs only.
    122.         detailed - Test entire matrix.
    123.         point    - Test with num_small small IOs, num_large large 
    124.                    IOs.
    125.         col      - Test a varying large IO load with num_small 
    126.                    small IOs.
    127.         row      - Test a varying small IO load with num_large 
    128.                    large IOs.
    129.         max      - Test varying loads up to the num_small and 
    130.                    num_large limits.
    131. ​
    132. hugenotneeded    Tells Orion to allocate regular pages when huge
    133.       pages are not available. With out this option, on systems
    134.       that support hugepages, Orion errors out when it cannot
    135.       allocate sufficient huge pages. -hugenotneeded is not set by
    136.       default.
    137. ​
    138. verbose      Prints tracing information to standard output if set.
    139.       Not set by default.
    140. datainput    Name of the input file for data to be written.
    141. ​
    142. is_lgwr      if set to true will run test using VIP level QOS, assuming the underlying transport supports it.
    143. ​
    144. ​
    145. Examples:
    146. For a preliminary set of data
    147.   -run simple 
    148. For a basic set of data
    149.   -run normal 
    150. To evaluate storage for an OLTP database
    151.   -run oltp 
    152. To evaluate storage for a data warehouse
    153.   -run dss 
    154. To generate combinations of 32KB and 1MB reads to random locations
    155.   -run advanced 
    156.   -size_small 32 -size_large 1024 -type rand 
    157.   -matrix detailed
    158. To generate multiple sequential 1MB write streams, simulating RAID0 striping
    159.   -run advanced 
    160.   -simulate RAID0 -write 100 -type seq
    161.   -matrix col -num_small 0
    162. ​

    配置lun测试文件

    1. #测试磁盘/dev/mapper/test1,/dev/mapper/test2,/dev/mapper/test3
    2. [root@rac1 tmp]# cat tlun.lun 
    3. /dev/mapper/test1
    4. /dev/mapper/test2
    5. /dev/mapper/test3

    开始进行IO性能测试

    1. #使用root用户
    2. export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1
    3. export PATH=$ORACLE_HOME/bin:$PATH:/usr/local/bin
    4. export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$ORACLE_HOME/rdbms/lib:/lib:/usr/lib
    5. export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib:$ORACLE_HOME/network/jlib
    6. #测试参数
    7. #模拟数据库随机小事务8K，读写占比7:3
    8. #每个节点开启2个压测，共四个压测
    9. /u01/app/oracle/product/11.2.0/db_1/bin/orion -run advanced -testname tlun  -num_disks 3 -size_small 8 -size_large 8 -type rand -write 30  -hugenotneeded
    10. ​
    11. [root@rac1 tmp]# /u01/app/oracle/product/11.2.0/db_1/bin/orion -run advanced -testname tlun  -num_disks 3 -size_small 8 -size_large 8 -type rand -write 30  -hugenotneeded
    12. ORION: ORacle IO Numbers -- Version 11.2.0.4.0
    13. tlun_20220904_1717
    14. Calibration will take approximately 23 minutes.
    15. Using a large value for -cache_size may take longer.
    16. ​
    17. ​
    18. Maximum Large MBPS=74.07 @ Small=0 and Large=6
    19. ​
    20. Maximum Small IOPS=9583 @ Small=6 and Large=0
    21. Small Read Latency: avg=623 us, min=85 us, max=10795 us, std dev=257 us @ Small=6 and Large=0
    22. Small Write Latency: avg=627 us, min=94 us, max=10431 us, std dev=243 us @ Small=6 and Large=0
    23. ​
    24. Minimum Small Latency=208 usecs @ Small=1 and Large=0
    25. Small Read Latency: avg=232 us, min=78 us, max=6918 us, std dev=138 us @ Small=1 and Large=0
    26. Small Write Latency: avg=151 us, min=88 us, max=4942 us, std dev=109 us @ Small=1 and Large=0
    27. Small Read / Write Latency Histogram @ Small=1 and Large=0 
    28.   Latency:    # of IOs (read)    # of IOs (write) 
    29.         0 - 1    us:    0      0
    30.         2 - 4    us:    0      0
    31.         4 - 8    us:    0      0
    32.         8 - 16    us:    0      0
    33.        16 - 32    us:    0      0
    34.        32 - 64    us:    0      0
    35.        64 - 128    us:    42901      42021
    36.       128 - 256    us:    98451      38491
    37.       256 - 512    us:    57190      4334
    38.       512 - 1024  us:    2413      549
    39.      1024 - 2048  us:    522      173
    40.      2048 - 4096  us:    126      37
    41.      4096 - 8192  us:    15      3
    42.      8192 - 16384  us:    0      0
    43.     16384 - 32768  us:    0      0
    44.     32768 - 65536  us:    0      0
    45.     65536 - 131072  us:    0      0
    46.    131072 - 262144  us:    0      0
    47.    262144 - 524288  us:    0      0
    48.    524288 - 1048576  us:    0      0
    49.   1048576 - 2097152  us:    0      0
    50.   2097152 - 4194304  us:    0      0
    51.   4194304 - 8388608  us:    0      0
    52.   8388608 - 16777216  us:    0      0
    53.  16777216 - 33554432  us:    0      0
    54.  33554432 - 67108864  us:    0      0
    55.  67108864 - 134217728  us:    0      0
    56. 134217728 - 268435456  us:    0      0
    57. ---测试结果
    58. Maximum Large MBPS=74.07
    59. Maximum Small IOPS=9583

    会在当前目录下生成测试的mbps,iops,latency的数据

    1. [root@rac1 tmp]# ls -lrt tlun*
    2. -rw-r--r-- 1 root root    54 Sep  4 17:13 tlun.lun
    3. -rw-r--r-- 1 root root 20240 Sep  4 17:38 tlun_20220904_1717_trace.txt
    4. -rw-r--r-- 1 root root   658 Sep  4 17:38 tlun_20220904_1717_mbps.csv
    5. -rw-r--r-- 1 root root   758 Sep  4 17:38 tlun_20220904_1717_lat.csv
    6. -rw-r--r-- 1 root root   734 Sep  4 17:38 tlun_20220904_1717_iops.csv
    7. -rw-r--r-- 1 root root 20587 Sep  4 17:38 tlun_20220904_1717_hist.txt
    8. -rw-r--r-- 1 root root  2151 Sep  4 17:38 tlun_20220904_1717_summary.txt

    总结:测试的结果为最大IOPS:9583,最大吞吐量74Mbps/s

    IO性能压测步骤(DBMS_RESOURCE_MANAGER.CALIBRATE_IO)

    DBMS_RESOURCE_MANAGER.CALIBRATE_IO为Oracle 11g数据库中的PL/SQL存储过程，可以用于对当前数据库实例的I/O性能进行测试，使用简单

    CALIBRATE_IO参数：

    进行IO性能压测

    1. set timing on
    2. set time on
    3. SET SERVEROUTPUT ON
    4. DECLARE
    5.   lat  INTEGER;
    6.   iops INTEGER;
    7.   mbps INTEGER;
    8. BEGIN
    9.   DBMS_RESOURCE_MANAGER.CALIBRATE_IO (num_physical_disks => 2,
    10.                    max_iops      => iops,
    11.                     max_mbps      => mbps,
    12.                     actual_latency   => lat
    13. );
    14.   DBMS_OUTPUT.PUT_LINE ('max_iops = ' || iops);
    15.   DBMS_OUTPUT.PUT_LINE ('latency  = ' || lat);
    16.   DBMS_OUTPUT.PUT_LINE ('max_mbps = ' || mbps);
    17. end;
    18. /

    查看IO性能压测结果

    1. SELECT TO_CHAR(start_time, 'DD-MON-YYY HH24:MI:SS') AS start_time,
    2.    TO_CHAR(end_time, 'DD-MON-YYY HH24:MI:SS') AS end_time,
    3.    max_iops,
    4.    max_mbps,
    5.    max_pmbps,
    6.    latency,
    7.    num_physical_disks AS disks
    8. FROM  dba_rsrc_io_calibrate;18:03:16 SQL> 18:03:16 SQL> 18:03:16   2  18:03:16   3  18:03:16   4  18:03:16   5  18:03:16   6  18:03:16   7  18:03:16   8  
    9. ​
    10. START_TIME         END_TIME     MAX_IOPS   MAX_MBPS  MAX_PMBPS    LATENCY      DISKS
    11. ---------------------- ---------------------- ---------- ---------- ---------- ---------- ----------
    12. 04-SEP-022 17:53:24    04-SEP-022 18:02:55     13070       1200     690         11     2

    总结:测试的结果为最大IOPS:13070,最大吞吐量1200Mbps/s

    IO性能压测步骤(FIO)

    fio软件安装

    1. tar xvf fio-2.17.tar.gz
    2. cd fio-3.13
    3. ./configure 
    4. make 
    5. make install

    确认安装成功

    1. [root@rac1 fio-2.17]# fio --help
    2. fio-2.17
    3. fio [options] [job options] <job file(s)>
    4.   --debug=options  Enable debug logging. May be one/more of:
    5.       process,file,io,mem,blktrace,verify,random,parse,
    6.       diskutil,job,mutex,profile,time,net,rate,compress,
    7.       steadystate,helperthread
    8.   --parse-only    Parse options only, don't start any IO
    9.   --output    Write output to file
    10.   --bandwidth-log  Generate aggregate bandwidth logs
    11.   --minimal    Minimal (terse) output
    12.   --output-format=type  Output format (terse,json,json+,normal)
    13.   --terse-version=type  Set terse version output format (default 3, or 2 or 4)
    14.   --version    Print version info and exit
    15.   --help    Print this page
    16.   --cpuclock-test  Perform test/validation of CPU clock
    17.   --crctest=type  Test speed of checksum functions
    18.   --cmdhelp=cmd    Print command help, "all" for all of them
    19.   --enghelp=engine  Print ioengine help, or list available ioengines
    20.   --enghelp=engine,cmd  Print help for an ioengine cmd
    21.   --showcmd    Turn a job file into command line options
    22.   --eta=when    When ETA estimate should be printed
    23.                 May be "always", "never" or "auto"
    24.   --eta-newline=time  Force a new line for every 'time' period passed
    25.   --status-interval=t  Force full status dump every 't' period passed
    26.   --readonly    Turn on safety read-only checks, preventing writes
    27.   --section=name  Only run specified section in job file, multiple sections can be specified
    28.   --alloc-size=kb  Set smalloc pool to this size in kb (def 16384)
    29.   --warnings-fatal  Fio parser warnings are fatal
    30.   --max-jobs=nr    Maximum number of threads/processes to support
    31.   --server=args    Start a backend fio server
    32.   --daemonize=pidfile  Background fio server, write pid to file
    33.   --client=hostname  Talk to remote backend(s) fio server at hostname
    34.   --remote-config=file  Tell fio server to load this local job file
    35.   --idle-prof=option  Report cpu idleness on a system or percpu basis
    36.       (option=system,percpu) or run unit work
    37.       calibration only (option=calibrate)
    38.   --inflate-log=log  Inflate and output compressed log
    39.   --trigger-file=file  Execute trigger cmd when file exists
    40.   --trigger-timeout=t  Execute trigger af this time
    41.   --trigger=cmd    Set this command as local trigger
    42.   --trigger-remote=cmd  Set this command as remote trigger
    43.   --aux-path=path  Use this path for fio state generated files
    44. ​
    45. Fio was written by Jens Axboe <jens.axboe@oracle.com>
    46.                    Jens Axboe <jaxboe@fusionio.com>
    47.                    Jens Axboe <axboe@fb.com>

    使用参数说明

    1. filename=/dev/mapper/test1 测试文件名称
    2. direct=1 绕过操作系统的buffer，即进行O_DIRECT操作
    3. iodepth  IO队列深度，在文件之上的保持的IO单元,默认对于每个文件来说是1，可以设置一个更大的值来提供并发度，iodepth大于1不会影响同步IO引擎(除非verify_async这个选项被设置)​
    4. thread   io默认会使用fork()创建job，如果这个选项设置的话，fio将使用pthread_create来创建线程
    5. rw=randwrite 测试随机写的I/O
    6. rw=randrw    测试随机写和读的I/O
    7. bs=8k 单次io的块文件大小为8k，仿照数据库的块大小
    8. size=1g 本次的测试文件大小为1g，以每次8k的io进行测试。
    9. numjobs=4 本次的测试线程为4.
    10. runtime=300 测试时间为300秒，如果不写测试时间，则将1g文件按每8k写完结束
    11. ioengine=psync io引擎使用pync方式
    12. rwmixwrite=30 在混合读写的模式下，写占30%
    13. rwmixread=70 在混合读写的模式下，读占70%
    14. group_reporting 关于显示结果的，汇总每个进程的信息​
    15. lockmem=1g 只使用1g内存进行测试​
    16. zero_buffers 用0初始化系统buffer​
    17. nrfiles=8 每个进程生成文件的数量

    IO性能压测

    1. ---数据库场景主要是测试随机读写
    2. ---混合随机读写测试
    3. [root@rac1 fio-2.17]# fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=8k -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest  -ioscheduler=deadline
    4. iotest: (g=0): rw=randrw, bs=8192B-8192B,8192B-8192B,8192B-8192B, ioengine=psync, iodepth=1
    5. ...
    6. fio-2.17
    7. Starting 4 threads
    8. fio: os or kernel doesn't support IO scheduler switching
    9. fio: os or kernel doesn't support IO scheduler switching
    10. fio: os or kernel doesn't support IO scheduler switching
    11. fio: os or kernel doesn't support IO scheduler switching
    12. #最大读写吞吐量/IOPS
    13. Jobs: 4 (f=4): [m(4)][100.0%][r=58.5MiB/s,w=25.4MiB/s][r=7486,w=3249 IOPS][eta 00m:00s]
    14. iotest: (groupid=0, jobs=4): err= 0: pid=97958: Sun Sep  4 19:05:36 2022
    15.    read: IOPS=6897, BW=53.1MiB/s (56.6MB/s)(2867MiB/53205msec)
    16.     clat (usec): min=75, max=16751, avg=391.74, stdev=335.46
    17.      lat (usec): min=75, max=16751, avg=392.36, stdev=335.54
    18.     clat percentiles (usec):
    19.      |  1.00th=[   80],  5.00th=[   92], 10.00th=[  108], 20.00th=[  147],
    20.      | 30.00th=[  205], 40.00th=[  266], 50.00th=[  322], 60.00th=[  382],
    21.      | 70.00th=[  454], 80.00th=[  548], 90.00th=[  724], 95.00th=[  924],
    22.      | 99.00th=[ 1656], 99.50th=[ 2096], 99.90th=[ 3312], 99.95th=[ 3760],
    23.      | 99.99th=[ 5984]
    24.   write: IOPS=2956, BW=23.2MiB/s (24.3MB/s)(1229MiB/53205msec)
    25.     clat (usec): min=82, max=16158, avg=412.63, stdev=334.53
    26.      lat (usec): min=83, max=16158, avg=414.12, stdev=334.91
    27.     clat percentiles (usec):
    28.      |  1.00th=[   89],  5.00th=[   96], 10.00th=[  110], 20.00th=[  149],
    29.      | 30.00th=[  219], 40.00th=[  290], 50.00th=[  354], 60.00th=[  418],
    30.      | 70.00th=[  494], 80.00th=[  588], 90.00th=[  756], 95.00th=[  940],
    31.      | 99.00th=[ 1592], 99.50th=[ 2064], 99.90th=[ 3312], 99.95th=[ 3824],
    32.      | 99.99th=[ 6304]
    33.     lat (usec) : 100=6.81%, 250=29.70%, 500=37.52%, 750=16.68%, 1000=5.18%
    34.     lat (msec) : 2=3.53%, 4=0.53%, 10=0.04%, 20=0.01%
    35.   cpu          : usr=1.31%, sys=19.35%, ctx=744673, majf=0, minf=9
    36.   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
    37.      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    38.      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    39.      issued rwt: total=366973,157315,0, short=0,0,0, dropped=0,0,0
    40.      latency   : target=0, window=0, percentile=100.00%, depth=1
    41. ​
    42. Run status group 0 (all jobs):
    43.    READ: bw=53.1MiB/s (56.6MB/s), 53.1MiB/s-53.1MiB/s (56.6MB/s-56.6MB/s), io=2867MiB (3006MB), run=53205-53205msec
    44.   WRITE: bw=23.2MiB/s (24.3MB/s), 23.2MiB/s-23.2MiB/s (24.3MB/s-24.3MB/s), io=1229MiB (1289MB), run=53205-53205msec
    45. ​
    46. ---混合顺序读写测试：
    47. [root@rac1 fio-2.17]# fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=rw -rwmixread=70 -ioengine=psync -bs=8k -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest  -ioscheduler=deadline
    48. iotest: (g=0): rw=rw, bs=8192B-8192B,8192B-8192B,8192B-8192B, ioengine=psync, iodepth=1
    49. ...
    50. fio-2.17
    51. Starting 4 threads
    52. fio: os or kernel doesn't support IO scheduler switching
    53. fio: os or kernel doesn't support IO scheduler switching
    54. fio: os or kernel doesn't support IO scheduler switching
    55. fio: os or kernel doesn't support IO scheduler switching
    56. Jobs: 4 (f=4): [M(4)][100.0%][r=48.5MiB/s,w=20.6MiB/s][r=6205,w=2627 IOPS][eta 00m:00s]
    57. iotest: (groupid=0, jobs=4): err= 0: pid=99479: Sun Sep  4 19:09:41 2022
    58.    read: IOPS=6943, BW=54.3MiB/s (56.9MB/s)(2867MiB/52851msec)
    59.     clat (usec): min=73, max=13036, avg=392.24, stdev=346.79
    60.      lat (usec): min=73, max=13036, avg=392.79, stdev=347.15
    61.     clat percentiles (usec):
    62.      |  1.00th=[   78],  5.00th=[   87], 10.00th=[  104], 20.00th=[  147],
    63.      | 30.00th=[  199], 40.00th=[  262], 50.00th=[  318], 60.00th=[  378],
    64.      | 70.00th=[  450], 80.00th=[  548], 90.00th=[  724], 95.00th=[  948],
    65.      | 99.00th=[ 1752], 99.50th=[ 2192], 99.90th=[ 3408], 99.95th=[ 4128],
    66.      | 99.99th=[ 5984]
    67.   write: IOPS=2976, BW=23.3MiB/s (24.4MB/s)(1229MiB/52851msec)
    68.     clat (usec): min=81, max=12001, avg=413.64, stdev=360.46
    69.      lat (usec): min=81, max=12001, avg=414.78, stdev=360.97
    70.     clat percentiles (usec):
    71.      |  1.00th=[   86],  5.00th=[   91], 10.00th=[  103], 20.00th=[  143],
    72.      | 30.00th=[  207], 40.00th=[  278], 50.00th=[  350], 60.00th=[  410],
    73.      | 70.00th=[  486], 80.00th=[  580], 90.00th=[  756], 95.00th=[  980],
    74.      | 99.00th=[ 1784], 99.50th=[ 2256], 99.90th=[ 3472], 99.95th=[ 4192],
    75.      | 99.99th=[ 6496]
    76.     lat (usec) : 100=8.62%, 250=28.16%, 500=37.90%, 750=15.82%, 1000=5.00%
    77.     lat (msec) : 2=3.81%, 4=0.63%, 10=0.06%, 20=0.01%
    78.   cpu          : usr=0.80%, sys=19.45%, ctx=750369, majf=0, minf=9
    79.   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
    80.      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    81.      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
    82.      issued rwt: total=366973,157315,0, short=0,0,0, dropped=0,0,0
    83.      latency   : target=0, window=0, percentile=100.00%, depth=1
    84. ​
    85. Run status group 0 (all jobs):
    86.    READ: bw=54.3MiB/s (56.9MB/s), 54.3MiB/s-54.3MiB/s (56.9MB/s-56.9MB/s), io=2867MiB (3006MB), run=52851-52851msec
    87.   WRITE: bw=23.3MiB/s (24.4MB/s), 23.3MiB/s-23.3MiB/s (24.4MB/s-24.4MB/s), io=1229MiB (1289MB), run=52851-52851msec
    88. ​
    89. ---其他场景
    90. ---随机读测试：
    91. fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=8k -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest 
    92. ​
    93. ---随机写测试：
    94. fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=8k -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest
    95. ​
    96. ---顺序读测试：
    97. fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=1M -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest
    98. ​
    99. ---顺序写测试：
    100. fio -filename=/dev/mapper/test1 -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=1M -size=1G -numjobs=4 -runtime=60 -group_reporting -name=iotest

    总结:

    以上内容为Oracle RAC集群网络，IO性能常用的测试工具以及测试方法，大家可以根据自身的环境以及使用场景进行选择。

  • 相关阅读:
    25. 答疑 - SAP OData 框架处理 Metadata 元数据请求的实现细节，前后端组件部署在同一台物理服务器
    superset study day01 (本地启动superset项目)
    三甲医院是什么？
    劲爆，Java 协程终于来了
    VS Code：CMake配置
    Excel表格密码保护，避免他人编辑
    【Golang开发面经】米哈游（一轮游）
    DASCTF X CBCTF 2022九月挑战赛 dino3d
    Android发展历程
    y91.第六章 微服务、服务网格及Envoy实战 -- 服务网格基础(二)
  • 原文地址：https://blog.csdn.net/sinat_36757755/article/details/126692594