apache stopped unexpectedly

遇到一台 centos 上的  apache (內建的 httpd) 不定時會關閉，還有重開機時固定會失敗。

Case 1：重開機時，apache 啟動失敗，查看 /var/log/httpd/error_log 檔案內容：

[Tue May 08 09:35:53.813466 2018] [mpm_prefork:notice] [pid 14255] AH00170: caught SIGWINCH, shutting down gracefully

Case 2：不定時 apache 會 crash, log 內容：

[Tue May 08 10:22:43.497635 2018] [suexec:notice] [pid 1973] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue May 08 10:22:43.498024 2018] [core:emerg] [pid 1973] (28)No space left on device: AH00023: Couldn't create the ssl-cache mutex 
AH00016: Configuration Failed
[Tue May 08 10:22:43.748373 2018] [suexec:notice] [pid 1977] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue May 08 10:22:43.748771 2018] [core:emerg] [pid 1977] (28)No space left on device: AH00023: Couldn't create the ssl-cache mutex 
AH00016: Configuration Failed
[Tue May 08 10:22:43.997656 2018] [suexec:notice] [pid 1981] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue May 08 10:22:43.998076 2018] [core:emerg] [pid 1981] (28)No space left on device: AH00023: Couldn't create the ssl-cache mutex 
AH00016: Configuration Failed

apache 掛掉後，使用 netstat -tulpn | grep LISTEN 指令，看不到有 port 被 listen.

使用 ps – aux | grep httpd 指令，也看不到有 process 在 memory 裡。

指令 service httpd status 的結果：

Redirecting to /bin/systemctl status httpd.service
● httpd.service - The Apache HTTP Server
 Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
 Active: failed (Result: start-limit) since Tue 2018-05-08 10:22:44 CST; 1h 19min ago
 Docs: man:httpd(8)
 man:apachectl(8)
 Process: 1981 ExecStart=/usr/sbin/httpd $OPTIONS (code=exited, status=1/FAILURE)
 Main PID: 1981 (code=exited, status=1/FAILURE)

May 08 10:22:44 localhost.localdomain systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
May 08 10:22:44 localhost.localdomain systemd[1]: Failed to start The Apache HTTP Server.
May 08 10:22:44 localhost.localdomain systemd[1]: Unit httpd.service entered failed state.
May 08 10:22:44 localhost.localdomain systemd[1]: httpd.service failed.
May 08 10:22:44 localhost.localdomain systemd[1]: httpd.service holdoff time over, scheduling restart.
May 08 10:22:44 localhost.localdomain systemd[1]: start request repeated too quickly for httpd.service
May 08 10:22:44 localhost.localdomain systemd[1]: Failed to start The Apache HTTP Server.
May 08 10:22:44 localhost.localdomain systemd[1]: Unit httpd.service entered failed state.
May 08 10:22:44 localhost.localdomain systemd[1]: httpd.service failed.

後來執行

 systemctl enable httpd

然後再 reboot 之後，居然就可以了，原來是 centos 的  service 需要額外多執行 enable 指令！

 

---

Apache Error Log:

log

No space left on device: AH00023: Couldn't create the ssl-cache mutex
AH00016: Configuration Failed

———

原因:

semaphores are used for communicating between the active processes of a certain application.

In the case of Apache, they’re used to communicate between the parent and child processes. If Apache can’t write these things down,

then it can’t communicate properly with all of the processes it starts.

i.e.

killall -9 httpd

———

解決方案:

[1]

/etc/sysctl.conf

kernel.msgmni = 512
kernel.sem = 250 128000 32 512

[2]

ipcs -s           # -s, –semaphores Write information about active semaphore sets.

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0x7a00c994 65536      zabbix     600        13
0x00000000 163841     apache     600        1
0x00000000 131074     apache     600        1

ipcrm -s N     # -s, –semaphore-id semid Remove the semaphore identified by semid

Batch Job

ipcs -s | grep apache | awk ‘{ print $2 }’ | xargs -n 1 ipcrm -s

---

安裝Apache ：
因為在 CentOS7 的 yum repository 中，Apache預設已安裝，版本是2.4版，如果其他的Linux Distribution，記得先確認一下版本，不同版本的設定檔是有些微差異性的。

httpd為Apache，mod_ssl則是進行網站ssl時需要使用

# yum install -y httpd mod_ssl

 

設定防火牆與SELinux：

# sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
# firewall-cmd --add-port=80/tcp --permanent
# firewall-cmd --add-port=443/tcp --permanent

 

配置Apache 2.4：

# vim /etc/httpd/conf/httpd.conf

說明：拿掉Indexes，主要防止目錄列出

Options Indexes FollowSymLinks
修改為
Options FollowSymLinks

 

參數說明：
AllowOverride ALL

ALL：
讓.htaccess管理全部
AuthConfig：
僅有網頁認證 (帳號密碼) 可覆寫
Indexes：
僅允許 Indexes 方面的覆寫；
Limits：
允許使用者利用 Allow, Deny 與 Order 管理可瀏覽的權限
None：
讓.htaccess 檔案無法使用

Require all granted

all granted：
允許各連線使用
all denied：
禁止被連線使用
ip 192.168.3.10：
僅允許IP 192.168.3.10 連線
not ip 192.168.3.10：
除了IP 192.168.3.10 之外，都可以連線
host kenwu0310.wordpress：
僅允許Hostname kenwu0310.wordpress連線
not host kenwu0310.wordpress：
除了Hostname kenwu0310.wordpress之外，都可以連線

 

設定apache開機啟動：

# systemctl enable httpd

重新啟動OS，因為前面SELinux需要關閉，也要確保之後重開機可以正確執行所有參數設定。

# reboot

---

如果你的 centos 裡有自訂的服務，如果使用 systemctl enable your-service-name 還是無去啟動的話，是因為你有缺少了相關檔案，需要補給 centos

I think CentOS 7 uses systemd. If that is the case for your system, try the following:

1. Put the script commands you wish to run in /usr/bin/myscript.
2. Remember to make the script executable with chmod +x.
3. Create the following file:

/usr/lib/systemd/system/my.service

[Unit]
Description=My Script

[Service]
Type=forking
ExecStart=/usr/bin/myscript

[Install]
WantedBy=multi-user.target

4. Reload the systemd service files: systemctl daemon-reload
5. Check that it is working with systemctl start my

---

如果上面的service 起不來，用這個指令看詳細內容：

systemctl status your_service_name.service

附註：上面的path 裡，除了 systemd 之外，還有一層 system.

附註：如果是直接在 ExecStart 裡跑 binary 有問題的話，請改成跑 shell script。這個原因是 Type 設定值造成，參考這一篇：
How can I start a systemd service only after mysql has started?
https://stackoverflow.max-everyday.com/2018/05/how-can-i-start-a-systemd-service-only-after-mysql-has-started/

 

如果還有問題就是需要增加這一個設定值：

WorkingDirectory=/home/your-path

 

---

修改 service 執行順序，使用After的設定值：

After a lot more reading I found the solution that works for me.

I read this guide, Running Services After the Network is up. A little quote from the guide:

This will ensure that all configured network devices are up and have an IP address assigned before boot continues.

This is exactly what I wanted so I enabled this service and set the a dependency rule in the service file for crond:

[root@srv]# systemctl enable NetworkManager-wait-online

[root@srv]# vi /lib/systemd/system/crond.service
  Requires=network.target
  After=syslog.target auditd.service systemd-user-sessions.service time-sync.target network.target mysqld.service

As mysqld is still based on the old init.d I needed to create a systemd service as suggested here, systemctl enable differs from systemctl start:

[root@srv]# vi /lib/systemd/system/mysqld.service
  [Unit]
  Description=MySQL Server
  After=network.target
  [Service]
  Type=forking
  ExecStart=/etc/rc.d/init.d/mysql start
  ExecStop=/etc/rc.d/init.d/mysql stop
  [Install]
  WantedBy=multi-user.target

[root@srv]# systemctl daemon-reload
[root@srv]# chkconfig mysql off
[root@srv]# systemctl enable mysqld

And finally setup the Apache service to startup after MySQL:

[root@srv]# vi /lib/systemd/system/httpd.service
  Requires=mysqld.service
  After=network.target remote-fs.target nss-lookup.target mysqld.service

This works for me at least.

I have used these commands to check it afterwards where I clearly can see that network is started up prior to at least MySQL and Apache. I though cannot see the crond anywhere but I can see it is working in my scripts:

[root@srv]# systemd-analyze critical-chain
  multi-user.target @10.510s
    + httpd.service @10.344s +165ms
      + mysqld.service @9.277s +1.065s
        + network.target @9.273s
          + network.service @8.917s +355ms
            + iptables.service @444ms +157ms
              + basic.target @443ms
                [CUT]

A couple of other useful commands I used are:

# See exactly what takes how long (who to blame for the delay)
[root@srv]# systemd-analyze blame

# Check available names that can be used in the service files
[root@srv]# systemctl list-unit-files

If any one can see any better way to do this then please share.