httpd stopping with no space warning but disks are half empty

[alanski]

httpd stopping with no space warning but disks are half empty.

Jun 10 15:09:07 : Failed to set a watch for httpd.service's PID file /run/apache2/httpd.pid: No space left on device

I have tried semaphore fixes (What is Semaphore? Why it's important for a Server Admin?)
This doesn't resolve the situation- only thing that works is a graceful reboot. However after ten minutes we lose the httpd service

When i check the apache log before the last time it shutdown it looks like this:

[Wed Jun 10 14:46:10.440751 2020] [suexec:notice] [pid 20930:tid 47067463844928] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jun 10 14:50:53.315067 2020] [:notice] [pid 2672:tid 47621486400576] ModSecurity for Apache/2.9.3 (http://www.modsecurity.org/) configured.
[Wed Jun 10 14:50:53.315085 2020] [:notice] [pid 2672:tid 47621486400576] ModSecurity: APR compiled version="1.7.0"; loaded version="1.7.0"
[Wed Jun 10 14:50:53.315093 2020] [:notice] [pid 2672:tid 47621486400576] ModSecurity: PCRE compiled version="8.32 "; loaded version="8.32 2012-11-30"
[Wed Jun 10 14:50:53.315097 2020] [:notice] [pid 2672:tid 47621486400576] ModSecurity: LUA compiled version="Lua 5.1"
[Wed Jun 10 14:50:53.315101 2020] [:notice] [pid 2672:tid 47621486400576] ModSecurity: YAJL compiled version="2.0.4"
[Wed Jun 10 14:50:53.315104 2020] [:notice] [pid 2672:tid 47621486400576] ModSecurity: LIBXML compiled version="2.9.7"
[Wed Jun 10 14:50:53.315107 2020] [:notice] [pid 2672:tid 47621486400576] ModSecurity: Status engine is currently disabled, enable it by set SecStatusEngine to On.
[Wed Jun 10 14:50:53.315661 2020] [suexec:notice] [pid 2672:tid 47621486400576] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Wed Jun 10 14:50:53.536844 2020] [mpm_worker:notice] [pid 2884:tid 47621486400576] AH00292: Apache/2.4.43 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 configured -- resuming normal operations
[Wed Jun 10 14:50:53.536900 2020] [core:notice] [pid 2884:tid 47621486400576] AH00094: Command line: '/usr/sbin/httpd'
[Wed Jun 10 14:55:29.730948 2020] [mpm_worker:notice] [pid 2884:tid 47621486400576] AH00297: SIGUSR1 received.  Doing graceful restart
[Wed Jun 10 14:55:29.855585 2020] [mpm_worker:notice] [pid 2884:tid 47621486400576] AH00292: Apache/2.4.43 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 configured -- resuming normal operations
[Wed Jun 10 14:55:29.855602 2020] [core:notice] [pid 2884:tid 47621486400576] AH00094: Command line: '/usr/sbin/httpd'
[Wed Jun 10 14:55:35.201834 2020] [mpm_worker:notice] [pid 2884:tid 47621486400576] AH00297: SIGUSR1 received.  Doing graceful restart
[Wed Jun 10 14:55:35.272259 2020] [mpm_worker:notice] [pid 2884:tid 47621486400576] AH00292: Apache/2.4.43 (cPanel) OpenSSL/1.1.1g mod_bwlimited/1.4 configured -- resuming normal operations
[Wed Jun 10 14:55:35.272275 2020] [core:notice] [pid 2884:tid 47621486400576] AH00094: Command line: '/usr/sbin/httpd'
[Wed Jun 10 14:55:59.998876 2020] [proxy_fcgi:error] [pid 2940:tid 47621821486848] (70014)End of file found: [client 195.152.29.238:33052] AH01075: Error dispatching request to : (reading input brigade)
[Wed Jun 10 14:56:03.348488 2020] [proxy_fcgi:error] [pid 2937:tid 47621808879360] (70014)End of file found: [client 195.152.29.238:33206] AH01075: Error dispatching request to : (reading input brigade)
[Wed Jun 10 14:56:06.815374 2020] [proxy_fcgi:error] [pid 2939:tid 47621810980608] (70014)End of file found: [client 195.152.29.238:33290] AH01075: Error dispatching request to : (reading input brigade)
[Wed Jun 10 14:59:46.363938 2020] [authz_core:error] [pid 4972:tid 47621817284352] [client 216.244.66.235:0] AH01630: client denied by server configuration: /home/wmddd/public_html/1-minute-tell-world-about-climate-change
[Wed Jun 10 14:59:46.364062 2020] [authz_core:error] [pid 4972:tid 47621817284352] [client 216.244.66.235:0] AH01630: client denied by server configuration: /home/wmddd/public_html/index.php
[Wed Jun 10 15:00:03.301741 2020] [mpm_worker:notice] [pid 2884:tid 47621486400576] AH00295: caught SIGTERM, shutting down

Urgently need a steer to investigate and resolve asap
tia

 

[GOT]

My go-to when I see disk full messages but df is reporting plenty of space is to check inodes

df -i

 

[cPanelLauren]

Not only check inodes which a lot of people don't know to check off hand (thanks @GOT ) but also what is the output of the following:

apachectl --status (when the issues occurs)
and
df -kh (anytime)