【存储】cosbench对象存储测试工具

查看下载下来log内容

1、main阶段failed

点击“view details”，找到Mission ID：

在cosbench安装目录的log/mission目录下，找到该mission id对应的日志文件，从日志文件中查看报错：

错误记录

Status Code: 500

Status Code: 500
 
2024-03-11 20:36:36,483 [ERROR] [AbstractOperator] - worker 13 fail to perform operation s3testqwer25
 
com.intel.cosbench.api.storage.StorageException: com.amazonaws.services.s3.model.AmazonS3Exception: We encountered an internal error, please try again.: cause(save bucket metadata to ns failed!s3testqwer25) (Service: Amazon S3; Status Code: 500; Error Code: InternalError; Request ID: 17BBB600BD8B7581), S3 Extended Request ID: 1a6d04ea-b557-4018-9ced-cc23e3798466
        at com.intel.cosbench.api.S3Stor.S3Storage.createContainer(S3Storage.java:149)
        at com.intel.cosbench.driver.operator.Preparer.doInit(Preparer.java:113)
        at com.intel.cosbench.driver.operator.Preparer.operate(Preparer.java:87)
        at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76)
        at com.intel.cosbench.driver.operator.Initializer.operate(Initializer.java:1)
        at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197)
        at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177)
        at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134)
        at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44)
        at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

原因
1,对象数据池没有创建

which: no nc in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)

如果不安装nc，会报错

which: no nc in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)

Ncat: Connection refused.

No appropriate tool found to detect cosbench driver status.

        如果不安装java，会报

Ncat: Connection refused.

启动失败

摘自：尝试在CentOS 7上启动cosbench遭遇的一些问题 - 之一 - 中道学友 - 博客园 (cnblogs.com)

尝试启动cosbench，遭遇报错：

[root]hopisdtmelabs14# cat driver-boot.log                                                                                                                                      /tmp/cosbench/0.4.2/log
java.net.BindException: Address already in use (Bind failed)
         at java.net.PlainSocketImpl.socketBind(Native Method)
         at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
         at java.net.ServerSocket.bind(ServerSocket.java:375)
         at java.net.ServerSocket.(ServerSocket.java:237)
         at java.net.ServerSocket.(ServerSocket.java:128)
         at org.eclipse.osgi.framework.internal.core.ConsoleManager.startConsole(ConsoleManager.java:186)
         at org.eclipse.osgi.framework.internal.core.ConsoleManager.startConsole(ConsoleManager.java:132)
         at org.eclipse.core.runtime.adaptor.EclipseStarter.startup(EclipseStarter.java:269)
         at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:175)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:498)
         at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:622)
         at org.eclipse.equinox.launcher.Main.basicRun(Main.java:577)
         at org.eclipse.equinox.launcher.Main.run(Main.java:1410)
         at org.eclipse.equinox.launcher.Main.main(Main.java:1386)
 
!SESSION 2020-06-03 06:04:23.609 -----------------------------------------------
 
eclipse.buildId=unknown
java.version=1.8.0_181
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Command-line arguments:  -console 18089
!ENTRY org.eclipse.osgi 4 0 2020-06-03 06:04:24.132
!MESSAGE Bundle plugins/cosbench-castor not found.

看起来像是网络端口18089已经被占用了，杀掉那些进程，问题解决。

如何找到占用18089端口的进程？举例如下：

# netstat -tulpn | grep 18089                                                                                                                                /tmp/cosbench/cos/conf
 
tcp6       0      0 :::18089                :::*                    LISTEN      4090/java
 
# ps aux | grep java                                                                                                                                         /tmp/cosbench/cos/conf
 
root      4090  0.0  0.4 21453452 315076 ?     Sl   10:39   0:18 java -Dcosbench.tomcat.config=conf/driver-tomcat-server.xml -server -cp main/org.eclipse.equinox.launcher_1.2.0.v20110502.jar org.eclipse.equinox.launcher.Main -configuration conf/.driver -console 18089
 
root      4822  0.3  0.3 20319980 262324 pts/0 Sl   21:13   0:07 java -Dcosbench.tomcat.config=conf/controller-tomcat-server.xml -server -cp main/org.eclipse.equinox.launcher_1.2.0.v20110502.jar org.eclipse.equinox.launcher.Main -configuration conf/.controller -console 19089
 
root      4965  0.0  0.0 112828  2316 pts/0    S+   21:46   0:00 grep --color=auto java

可以看出来在上面的例子中，4090就是占用了18089端口的进程号了，通过ps aux 还能看到详细的命令行参数。

但cosbench driver 还是启动不了。

[root]hopisdtmelabs14# bash ./start-driver.sh                                                                                                                                          /tmp/cosbench/cos
 
Launching osgi framwork ...
Successfully launched osgi framework!
Booting cosbench driver ...
which: no nc in (/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/usr/local/tme/bin:/usr/local/tme/tme_portal/perf_web/bin)
No appropriate tool found to detect cosbench driver status.

看报错，应该是nc（ncat）这个软件没有安装。可是通过yum还没法正常安装。我通过下面的命令解决了这个问题：

​
# yum -y install wget
# wget  https://nmap.org/dist/ncat-7.80-1.x86_64.rpm
# yum localinstall ncat-7.80-1.x86_64.rpm
# cd /usr/bin
# ln -s ncat nc

装好了nc之后，还是有错误。

# bash ./start-driver.sh                                                                                                                                          /tmp/cosbench/cos
 
Launching osgi framwork ...
Successfully launched osgi framework!
Booting cosbench driver ...
...........................................................
Starting    cosbench-log_0.4.2    [ERROR]
...........................................................
Starting    cosbench-tomcat_0.4.2    [ERROR]
..........................................................
Starting    cosbench-config_0.4.2    [ERROR]
...........................................................
Starting    cosbench-http_0.4.2    [ERROR]

我怀疑是我的java的版本过高，所以卸掉java 8， 安装java 7。

# rpm -aq | grep -i jdk                                                                                                                                           /tmp/cosbench/cos
 
java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64
copy-jdk-configs-3.3-10.el7_5.noarch
java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64
 
 
# rpm -e java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64
# rpm -e java-1.8.0-openjdk-headless-1.8.0.181-3.b13.el7_5.x86_64
# yum remove java

安装java 7.

#yum install java-1.7.0-openjdk

结果遇到了报错，说有两个软件包太旧了，达不到安装的要求。

尝试安装更新，结果还是报错，无法更新。

[Errno 14] HTTP/HTTPS Error 404

于是执行下面的命令，对yum进行修复。

#yum clean all

#rm -rf /var/cache/yum/*

之后java 7也成功安装了。

但还是报错：

# cat driver-boot.log                                                                                                                                         /tmp/cosbench/cos/log
Listening on port 0.0.0.0/0.0.0.0:18089 ...
!SESSION 2020-06-03 10:12:59.683 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.7.0_261
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Command-line arguments:  -console 18089
!ENTRY org.eclipse.osgi 4 0 2020-06-03 10:13:00.367
!MESSAGE Bundle plugins/cosbench-castor not found.
!ENTRY org.eclipse.osgi 4 0 2020-06-03 10:13:00.368
!MESSAGE Bundle plugins/cosbench-log4j not found.
!ENTRY org.eclipse.osgi 4 0 2020-06-03 10:13:00.368
!MESSAGE Bundle plugins/cosbench-log@6:start not found.
!ENTRY org.eclipse.osgi 4 0 2020-06-03 10:13:00.369
!MESSAGE Bundle plugins/cosbench-config@6:start not found.
 
…
 
…

Github的cosbench的站点里的issue#383里的步骤（具体步骤描述在这篇文章），即删掉cosbench-start.sh里的一行（如下），我也试过了，并没有解决问题。

TOOL_PARAMS="-i 0" -----Delete this line

经过查询，找到了这篇文章，推荐不要使用正式版的0.4.2，推荐试试0.4.2.c4, 于是下载了更早的一个版本0.4.2.c4。

问题解决了，终于成功启动了cosbench driver。

# bash ./start-driver.sh                                                                                                                                     /tmp/cosbench/0.4.2.c4
Launching osgi framwork ...
Successfully launched osgi framework!
Booting cosbench driver ...
.
Starting    cosbench-log_0.4.2    [OK]
.
Starting   cosbench-tomcat_0.4.2    [OK]
Starting    cosbench-config_0.4.2    [OK]
Starting    cosbench-http_0.4.2    [OK]
Starting    cosbench-cdmi-util_0.4.2    [OK]
Starting    cosbench-core_0.4.2    [OK]
Starting    cosbench-core-web_0.4.2    [OK]
Starting    cosbench-api_0.4.2    [OK]
Starting    cosbench-mock_0.4.2    [OK]
Starting    cosbench-ampli_0.4.2    [OK]
Starting    cosbench-swift_0.4.2    [OK]
Starting    cosbench-keystone_0.4.2    [OK]
Starting    cosbench-httpauth_0.4.2    [OK]
Starting    cosbench-s3_0.4.2    [OK]
Starting    cosbench-librados_0.4.2    [OK]
Starting    cosbench-scality_0.4.2    [OK]
Starting    cosbench-cdmi-swift_0.4.2    [OK]
Starting    cosbench-cdmi-base_0.4.2    [OK]
Starting    cosbench-driver_0.4.2    [OK]
Starting    cosbench-driver-web_0.4.2    [OK]
Successfully started cosbench driver!
Listening on port 0.0.0.0/0.0.0.0:18089 ...
Persistence bundle starting...
Persistence bundle started.
----------------------------------------------
!!! Service will listen on web port: 18088 !!!
----------------------------------------------

不知道为什么，在CentOS 7上运行0.4.2 明显的有问题，却还不被修复，结果还要去使用稍早的版本release candidate 4, 这个事儿弄的挺恶心的。

To the English readers:
I document the process and issues here in detail, hope other people hit on the same issue can see it, and save time.

参考资料

 
================
 
How to check if port is in use on Linux or Unix
https://www.cyberciti.biz/faq/unix-linux-check-if-port-is-in-use-command/
NETCAT THE MULTI-PURPOSE NETWORKING TOOL
landoflinux.com
Uninstall JDK rpm to reinstall
centos - Uninstall JDK rpm to reinstall - Unix & Linux Stack Exchange
yum-errors
https://wiki.centos.org/yum-errors
CosBench on Red Hat Linux
http://www.vtechdiary.com/2018/09/cosbench-on-red-hat-linux.html
Can't start the system #380
Can't start the system · Issue #380 · intel-cloud/cosbench · GitHub
 
​

参考资料

使用COSBench工具對ceph s3接口進行壓力測試--續 - JavaShuo

尝试在CentOS 7上启动cosbench遭遇的一些问题 - 之一 - 中道学友 - 博客园 (cnblogs.com)