测试环境:
Server version: Apache/2.4.6 (CentOS)
Server built: Nov 10 2021 14:26:31
注意,httpd的版本号是2.4.6。之前重启都正常的,突然就不能启动了,查看httpd的日志:/var/log/httpd/error.log日志,发现如下错误信息:
[Wed Aug 17 15:43:26.756580 2022] [suexec:notice] [pid 31792] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Aug 17 15:43:26.757096 2022] [core:emerg] [pid 31792] (28)No space left on device: AH00023: Couldn't create the ssl-cache mutex
AH00016: Configuration Failed
如果不想继续了解后续的内容,则执行以下语句,问题即可解决:
ipcs -s | grep apache | awk '{ print $2 }' | xargs -n 1 ipcrm -s
解释:
ipcs: 该命令用于检查多进程间通信,包括共享内存、消息队列、信号量等。
ipcrm:该命令则用于删除一个或更多的消息队列、信号量集或者共享内存标识。
那么,总结一句就是查出apache进程尚未释放的信号量,然后将其删除。
怀着好奇,我们先不执行执行上面的命令,先研究下错误是如何抛出来的。
执行ipcs命令,至于为什么要执行这个命令,我也不懂,网上查的,以后有机会再深入了解吧。
ipcs
查看输出的内容,如下:
- [root@localhost /root]# ipcs
-
- ------ Message Queues --------
- key msqid owner perms used-bytes messages
-
- ------ Shared Memory Segments --------
- key shmid owner perms bytes nattch status
- 0x0113aa67 393216 root 600 1000 0
- 0x0113ec74 425985 root 600 1000 0
- 0x01138f71 458754 root 600 1000 0
- 0x01130890 688131 root 600 1000 0
- 0x01133bd1 753668 root 600 1000 0
- 0x011319cc 983045 root 600 1000 0
- 0x0113e57e 1015814 root 600 1000 0
- 0x0113b4e8 1114119 root 600 1000 0
- 0x0113eede 1146888 root 600 1000 0
- 0x0113385b 1179657 root 600 1000 0
- 0x0113ddcf 1212426 root 600 1000 0
- 0x011330a7 1409035 root 600 1000 0
- 0x0113aef1 1441804 root 600 1000 0
- 0x0113f883 1474573 root 600 1000 0
- 0x01137a77 1835022 root 600 1000 0
- 0x0113a3f7 2949135 root 600 1000 0
- 0x0113d3ff 3178512 root 600 1000 0
- 0x0113bc22 3342353 root 600 1000 0
- 0x0113cffb 3375122 root 600 1000 0
- 0x0113fd23 3407891 root 600 1000 0
- 0x0113054e 3833876 root 600 1000 0
-
- ------ Semaphore Arrays --------
- key semid owner perms nsems
- 0x00000000 3211264 apache 600 1
- 0x00000000 3244033 apache 600 1
- 0x00000000 3178498 apache 600 1
- 0x00000000 3276803 apache 600 1
- 0x00000000 3309572 apache 600 1
- 0x00000000 3342341 apache 600 1
- 0x00000000 3473414 apache 600 1
- 0x00000000 3506183 apache 600 1
- 0x00000000 3440648 apache 600 1
- 0x00000000 3538953 apache 600 1
- 0x00000000 3571722 apache 600 1
- 0x00000000 3604491 apache 600 1
- 0x00000000 3735564 apache 600 1
- 0x00000000 3768333 apache 600 1
- 0x00000000 3702798 apache 600 1
- 0x00000000 3801103 apache 600 1
- 0x00000000 3833872 apache 600 1
- 0x00000000 3866641 apache 600 1
- 0x00000000 5570578 apache 600 1
- 0x00000000 5603347 apache 600 1
- 0x00000000 5537812 apache 600 1
- 0x00000000 5636117 apache 600 1
- 0x00000000 5668886 apache 600 1
- 0x00000000 5701655 apache 600 1
- 0x00000000 6094872 apache 600 1
- 0x00000000 6127641 apache 600 1
- 0x00000000 6062106 apache 600 1
- 0x00000000 6160411 apache 600 1
- 0x00000000 6193180 apache 600 1
- 0x00000000 6225949 apache 600 1
- 0x00000000 7831582 apache 600 1
- 0x00000000 7864351 apache 600 1
- 0x00000000 7798816 apache 600 1
- 0x00000000 7897121 apache 600 1
- 0x00000000 7929890 apache 600 1
- 0x00000000 7962659 apache 600 1
- 0x00000000 8093732 apache 600 1
- 0x00000000 8126501 apache 600 1
- 0x00000000 8060966 apache 600 1
- 0x00000000 8159271 apache 600 1
- 0x00000000 8192040 apache 600 1
- 0x00000000 8224809 apache 600 1
- 0x00000000 8880170 apache 600 1
- 0x00000000 8912939 apache 600 1
- 0x00000000 8847404 apache 600 1
- 0x00000000 8945709 apache 600 1
- 0x00000000 8978478 apache 600 1
- 0x00000000 9011247 apache 600 1
- 0x00000000 9142320 apache 600 1
- 0x00000000 9175089 apache 600 1
- 0x00000000 9109554 apache 600 1
- 0x00000000 9207859 apache 600 1
- 0x00000000 9240628 apache 600 1
- 0x00000000 9273397 apache 600 1
- 0x00000000 9404470 apache 600 1
- 0x00000000 9437239 apache 600 1
- 0x00000000 9371704 apache 600 1
- 0x00000000 9470009 apache 600 1
- 0x00000000 9502778 apache 600 1
- 0x00000000 9535547 apache 600 1
- 0x00000000 9666620 apache 600 1
- 0x00000000 9699389 apache 600 1
- 0x00000000 9633854 apache 600 1
- 0x00000000 9732159 apache 600 1
- 0x00000000 9764928 apache 600 1
- 0x00000000 9797697 apache 600 1
- 0x00000000 11141186 apache 600 1
- 0x00000000 11173955 apache 600 1
- 0x00000000 11108420 apache 600 1
- 0x00000000 11206725 apache 600 1
- 0x00000000 11239494 apache 600 1
- 0x00000000 11272263 apache 600 1
- 0x00000000 11403336 apache 600 1
- 0x00000000 11436105 apache 600 1
- 0x00000000 11370570 apache 600 1
- 0x00000000 11468875 apache 600 1
- 0x00000000 11501644 apache 600 1
- 0x00000000 11534413 apache 600 1
- 0x00000000 11665486 apache 600 1
- 0x00000000 11698255 apache 600 1
- 0x00000000 11632720 apache 600 1
- 0x00000000 11731025 apache 600 1
- 0x00000000 11763794 apache 600 1
- 0x00000000 11796563 apache 600 1
- 0x00000000 14450772 apache 600 1
- 0x00000000 14483541 apache 600 1
- 0x00000000 14418006 apache 600 1
- 0x00000000 14516311 apache 600 1
- 0x00000000 14549080 apache 600 1
- 0x00000000 14581849 apache 600 1
- 0x00000000 23068762 apache 600 1
- 0x00000000 23101531 apache 600 1
- 0x00000000 23035996 apache 600 1
- 0x00000000 23134301 apache 600 1
- 0x00000000 23167070 apache 600 1
- 0x00000000 23199839 apache 600 1
- 0x00000000 24805472 apache 600 1
- 0x00000000 24838241 apache 600 1
- 0x00000000 24772706 apache 600 1
- 0x00000000 24871011 apache 600 1
- 0x00000000 24903780 apache 600 1
- 0x00000000 24936549 apache 600 1
- 0x00000000 26116198 apache 600 1
- 0x00000000 26148967 apache 600 1
- 0x00000000 26083432 apache 600 1
- 0x00000000 26181737 apache 600 1
- 0x00000000 26214506 apache 600 1
- 0x00000000 26247275 apache 600 1
- 0x00000000 26378348 apache 600 1
- 0x00000000 26411117 apache 600 1
- 0x00000000 26345582 apache 600 1
- 0x00000000 26443887 apache 600 1
- 0x00000000 26476656 apache 600 1
- 0x00000000 26509425 apache 600 1
- 0x00000000 26640498 apache 600 1
- 0x00000000 26673267 apache 600 1
- 0x00000000 26607732 apache 600 1
- 0x00000000 26706037 apache 600 1
- 0x00000000 26738806 apache 600 1
- 0x00000000 26771575 apache 600 1
- 0x00000000 29851768 apache 600 1
- 0x00000000 29884537 apache 600 1
- 0x00000000 29819002 apache 600 1
- 0x00000000 29917307 apache 600 1
- 0x00000000 29950076 apache 600 1
- 0x00000000 29982845 apache 600 1
通过上面的输出内容,是不是发现有很多apache进程对应的信号量(Semaphore)。
再统计一下,apache的Semaphore共有多少个:
- [root@localhost /root]# ipcs | grep apache | wc -l
- 126
输出126,但这个数字又能说明什么呢,系统默认的上限又是多少呢,带着这个疑问,输入以下命令,获取系统的默认设置:
ipcs -l
执行结果如下:
- [root@localhost /root]# ipcs -l
-
- ------ Messages Limits --------
- max queues system wide = 32768
- max size of message (bytes) = 8192
- default max size of queue (bytes) = 16384
-
- ------ Shared Memory Limits --------
- max number of segments = 4096
- max seg size (kbytes) = 18014398509465599
- max total shared memory (kbytes) = 18014398442373116
- min seg size (bytes) = 1
-
- ------ Semaphore Limits --------
- max number of arrays = 128
- max semaphores per array = 250
- max semaphores system wide = 32000
- max ops per semop call = 32
- semaphore max value = 32767
根据上下文对比,我们只需要关心Semaphore Limits 这部分,从上图可以看到,Semaphore Arrays 默认的上限是128,前面我们统计出来已经存在126个了。因此,大胆推测,启动httpd的时候新增的Semaphore Arrays 超过了128,导致错误出现。
接下来,我们验证一下启动httpd之后,Semaphore Arrays一共会有多少个。启动之前,需要执行前面所提的命令,先将所有的Semaphore清除掉才能启动成功。
启动成功之后,查看httpd的状态:
- [root@localhost ~]# systemctl start httpd
- [root@localhost ~]# systemctl status httpd
- ● httpd.service - The Apache HTTP Server
- Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
- Active: active (running) since Thu 2022-07-21 18:17:27 CST; 3 weeks 6 days ago
- Docs: man:httpd(8)
- man:apachectl(8)
- Main PID: 20907 (httpd)
- Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
- CGroup: /system.slice/httpd.service
- ├─ 7453 /usr/sbin/httpd -DFOREGROUND
- ├─ 7454 /usr/sbin/httpd -DFOREGROUND
- ├─ 7455 /usr/sbin/httpd -DFOREGROUND
- ├─ 7456 /usr/sbin/httpd -DFOREGROUND
- ├─ 7457 /usr/sbin/httpd -DFOREGROUND
- └─20907 /usr/sbin/httpd -DFOREGROUND
-
-
从上面的输出可以看到,启动之后,开启了6个进程,一个父进程(pid=20907),5个子进程。也可以通过ps -ef|grep httpd 命令查看,得到的还是6个进程。
- [root@localhost ~]# ps -ef|grep httpd
- apache 7453 20907 0 Aug14 ? 00:00:01 /usr/sbin/httpd -DFOREGROUND
- apache 7454 20907 0 Aug14 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
- apache 7455 20907 0 Aug14 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
- apache 7456 20907 0 Aug14 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
- apache 7457 20907 0 Aug14 ? 00:00:00 /usr/sbin/httpd -DFOREGROUND
- root 8753 3688 0 19:10 pts/0 00:00:00 grep --color=auto httpd
- root 20907 1 0 Jul21 ? 00:01:39 /usr/sbin/httpd -DFOREGROUND
好,接下来,再用命令ipcs查看一下新创建的Semaphore总数,也可以使用ipcs -s,只看Semaphore部分:
- [root@localhost ~]# ipcs
-
- ------ Message Queues --------
- key msqid owner perms used-bytes messages
-
- ------ Shared Memory Segments --------
- key shmid owner perms bytes nattch status
- 0xe8136391 262144 root 600 1000 0
- 0xf513d5ac 1179649 root 600 1000 0
- 0x8f13cabc 1409026 root 600 1000 6
- 0x00000072 1441795 root 444 1 0
-
- ------ Semaphore Arrays --------
- key semid owner perms nsems
- 0x00000000 1867776 apache 600 1
- 0x00000000 1900545 apache 600 1
- 0x00000000 1343490 apache 600 1
- 0x00000000 1933315 apache 600 1
- 0x00000000 1966084 apache 600 1
- 0x00000000 1998853 apache 600 1
从上图可知,启动httpd之后,Semaphore 总数为6,和进程个数一致。当我们将httpd停止后,再查看一下Semaphore的总数,我所在的测试机器上,总数并未变为0,还是6个。当我再次重启httpd的时候,Semaphore的总数还是6,且内容也没有变化。那么,什么时候它才会自动清除呢,这里留下了疑问,待以后有了解再来补充吧。
到这里,疑问又来了,既然停止了再重启,Semaphore的内容并未变化,那什么情况下才会出现超过128个Semaphore的情况呢?
另外,我测试了httpd的2.4.53版本,它的情况又有两处不同:
那么, 是从某个版本开始就不涉及Semaphore的问题了?
详细内容如下:
- [root@localhost mongo]# httpd -v
- Server version: Apache/2.4.53 (Unix)
- Server built: May 10 2022 18:55:44
- [root@localhost mongo]#
- [root@localhost mongo]#
- [root@localhost mongo]# systemctl status httpd
- ● httpd.service - LSB: start and stop Apache HTTP Server
- Loaded: loaded (/etc/rc.d/init.d/httpd; bad; vendor preset: disabled)
- Active: active (running) since Thu 2022-07-21 18:14:40 CST; 3 weeks 6 days ago
- Docs: man:systemd-sysv-generator(8)
- Main PID: 22906 (httpd)
- CGroup: /system.slice/httpd.service
- ├─ 9671 /usr/sbin/httpd
- ├─ 9672 /usr/sbin/httpd
- ├─ 9673 /usr/sbin/httpd
- └─22906 /usr/sbin/httpd
-
- Jul 21 18:14:40 localhost.localdomain systemd[1]: Starting LSB: start and stop Apache HTTP Server...
- Jul 21 18:14:40 localhost.localdomain httpd[22899]: Starting httpd: AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerN...s this message
- Jul 21 18:14:40 localhost.localdomain httpd[22899]: [ OK ]
- Jul 21 18:14:40 localhost.localdomain systemd[1]: Started LSB: start and stop Apache HTTP Server.
- Hint: Some lines were ellipsized, use -l to show in full.
- [root@localhost mongo]#
- [root@localhost mongo]#
- [root@localhost mongo]# ipcs
-
- ------ Message Queues --------
- key msqid owner perms used-bytes messages
-
- ------ Shared Memory Segments --------
- key shmid owner perms bytes nattch status
- 0x00000072 0 root 444 1 0
-
- ------ Semaphore Arrays --------
- key semid owner perms nsems
-
- [root@localhost mongo]#
疑问越来越多,带着疑问结束本文了,等有这方面的储备了再回来补充吧!