The service “httpd” appears to be down.

[DarkxPunk]

Woke up this morning to countless emails telling me the httpd service appears to be done. I am not understanding what is going wrong in the logs and not sure exactly what to share as nothing calls out as being a problem. At first I couldn't access any web pages, could access via FTP and made sure to get a copy of the system and user backups. I could not access WHM but did notice in the email that one of my users had excessive usage, after suspending that user I was able to access WHM again and restart the server. After the restart everything seemed to work but about a minute later I could not access any webpages again. Email, ftp, ssh, everything else is functioning normally.

This was in the startup log from the email:

Apr 03 17:19:07 vps2.ormt.ca systemd[1]: Starting Apache web server managed by cPanel EasyApache...
Apr 03 17:19:08 vps2.ormt.ca systemd[1]: Can't open PID file /run/apache2/httpd.pid (yet?) after start: No such file or directory
Apr 03 17:19:09 vps2.ormt.ca systemd[1]: Started Apache web server managed by cPanel EasyApache.

However I can affirm /run/apache2/httpd.pid exists.

Any guidance or what logs I can provide to get some clarity, let me know. Been down all day...

 

[quietFinn]

Maybe this helps:

How to diagnose high server loads

What is considered high memory, high CPU, or high I/O? You will want first to determine it is related to memory, CPU, or I/O; you can do this with the sar command. Before you understand the sar o...

support.cpanel.net

 

[DarkxPunk]

Maybe this helps:

How to diagnose high server loads

What is considered high memory, high CPU, or high I/O? You will want first to determine it is related to memory, CPU, or I/O; you can do this with the sar command. Before you understand the sar o...

support.cpanel.net

It doesn't seem load is the issue unless I am missing something, as you can see in the attached image.

 

[cPRex]

The best thing to do in this scenario is likely to check the Apache logs from around the time you received the message. Those can be found at /etc/apache2/logs/error_log in SSH on the system. Can you see if there is anything useful there?

 

[DarkxPunk]

This is what I find around the same time as the first email.

[Sat Apr 02 19:23:35.715145 2022] [:error] [pid 12401:tid 46957607937792] [client 204.12.215.61:62265] [client 204.12.215.61] ModSecurity: Warning. Operator EQ matched 0 at REQUEST_HEADERS. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "368"] [id "920340"] [rev "3"] [msg "Request Containing Content, but Missing Content-Type header"] [severity "NOTICE"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [hostname "<removed>"] [uri "/xmlrpc.php"] [unique_id "YkkFJ62Zt0YdcftWQF9oMAAAAEA"], referer: Google
[Sat Apr 02 19:55:41.798324 2022] [mpm_worker:notice] [pid 6853:tid 46957366084672] AH00295: caught SIGTERM, shutting down
[Sat Apr 02 19:55:43.617700 2022] [core:notice] [pid 20345:tid 47529501979712] SELinux policy enabled; httpd running as context system_u:system_r:unconfined_service_t:s0
[Sat Apr 02 19:55:43.643244 2022] [ssl:warn] [pid 20345:tid 47529501979712] AH01909: <removed>:443:0 server certificate does NOT include an ID which matches the server name
[Sat Apr 02 19:55:43.713275 2022] [:notice] [pid 20345:tid 47529501979712] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Sat Apr 02 19:55:43.713321 2022] [:notice] [pid 20345:tid 47529501979712] ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0"
[Sat Apr 02 19:55:43.713348 2022] [:notice] [pid 20345:tid 47529501979712] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Sat Apr 02 19:55:43.713370 2022] [:notice] [pid 20345:tid 47529501979712] ModSecurity: LUA compiled version="Lua 5.1"
[Sat Apr 02 19:55:43.713391 2022] [:notice] [pid 20345:tid 47529501979712] ModSecurity: YAJL compiled version="2.0.4"
[Sat Apr 02 19:55:43.713412 2022] [:notice] [pid 20345:tid 47529501979712] ModSecurity: LIBXML compiled version="2.9.7"
[Sat Apr 02 19:55:43.713433 2022] [:notice] [pid 20345:tid 47529501979712] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Sat Apr 02 19:55:43.715609 2022] [suexec:notice] [pid 20345:tid 47529501979712] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Sat Apr 02 19:55:43.986689 2022] [ssl:warn] [pid 20348:tid 47529501979712] AH01909: <removed>:443:0 server certificate does NOT include an ID which matches the server name
[Sat Apr 02 19:55:44.097862 2022] [mpm_worker:notice] [pid 20348:tid 47529501979712] AH00292: Apache/2.4.53 (cPanel) OpenSSL/1.1.1n mod_bwlimited/1.4 configured -- resuming normal operations
[Sat Apr 02 19:55:44.098095 2022] [core:notice] [pid 20348:tid 47529501979712] AH00094: Command line: '/usr/sbin/httpd'
[Sat Apr 02 19:55:50.108430 2022] [mpm_worker:error] [pid 20348:tid 47529501979712] AH00287: server is within MinSpareThreads of MaxRequestWorkers, consider raising the MaxRequestWorkers setting
[Sat Apr 02 19:55:54.113896 2022] [mpm_worker:error] [pid 20348:tid 47529501979712] AH00286: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[Sat Apr 02 20:11:42.562237 2022] [mpm_worker:notice] [pid 20348:tid 47529501979712] AH00295: caught SIGTERM, shutting down

 

[cPRex]

Thanks for posting that - unfortunately it doesn't tell us much as it just shows a "sigterm" and then the process restarts.

What does this command show?

grep "server reached MaxRequestWorkers" /etc/apache2/logs/error_log | wc -l

I would expect the output to be a whole number.

 

[DarkxPunk]

Yes it did, it said 180.

 

[cPRex]

Alrighty...and one more thing. If you run this:

head /etc/apache2/logs/error_log

how far back does the log go?

 

[DarkxPunk]

[Tue Feb 09 13:35:23.035298 2021] [core:notice] [pid 14831] SELinux policy enabled; httpd running as context system_u:system_r:unconfined_service_t:s0
[Tue Feb 09 13:35:23.037378 2021] [ssl:warn] [pid 14831] AH01909: <removed>:443:0 server certificate does NOT include an ID which matches the server name
[Tue Feb 09 13:35:23.037997 2021] [ssl:warn] [pid 14831] AH01909: <removed>:443:0 server certificate does NOT include an ID which matches the server name
[Tue Feb 09 13:35:23.038220 2021] [:notice] [pid 14831] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Tue Feb 09 13:35:23.038227 2021] [:notice] [pid 14831] ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0"
[Tue Feb 09 13:35:23.038236 2021] [:notice] [pid 14831] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Tue Feb 09 13:35:23.038242 2021] [:notice] [pid 14831] ModSecurity: LUA compiled version="Lua 5.1"
[Tue Feb 09 13:35:23.038246 2021] [:notice] [pid 14831] ModSecurity: YAJL compiled version="2.0.4"
[Tue Feb 09 13:35:23.038251 2021] [:notice] [pid 14831] ModSecurity: LIBXML compiled version="2.9.7"
[Tue Feb 09 13:35:23.038256 2021] [:notice] [pid 14831] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.

That is what I get.

 

[cPRex]

Alright, so over a year. I was wondering if the server was reaching the MaxRequestWorkers value so frequently that the server monitoring tools interpreted that as the service being offline, but that doesn't seem to be the case.

With the details I have here I'm really not sure what may have happened. You're always welcome to open a ticket with our team so we can take a look at the system directly.

 

[DarkxPunk]

So my server host investigated the issue and discovered it was an excess of apache sessions and is associating it with a DDoS attack. Installed ConfigServ Firewall (CSF) as a protection. Seems to have resolved the issue so far. Suppose I will need to now deal with the prior issues I had with CSF. Please mark as resolved.

 

[quietFinn]

So my server host investigated the issue and discovered it was an excess of apache sessions and is associating it with a DDoS attack. Installed ConfigServ Firewall (CSF) as a protection. Seems to have resolved the issue so far. Suppose I will need to now deal with the prior issues I had with CSF. Please mark as resolved.

cPanel forums is the best place to ask if you have problems with CSF/LFD.