Apache keeps crashing: No space left on device.

Go3team

Member
Dec 4, 2003
12
0
151
This happens at random, but when it does, the automatic restart feature will not work. I have to go in, and restart via the command line. Here is the log:

[Fri Apr 22 11:47:07 2005] [warn] child process 16194 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 16195 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 16197 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 16198 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 16200 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 16204 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 15414 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 15419 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 15420 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 20045 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 20287 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 20314 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:07 2005] [warn] child process 20317 still did not exit, sending a SIGTERM
[Fri Apr 22 11:47:10 2005] [warn] pid file /usr/local/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
semget: No space left on device

I did not post the remaining lines as they are identical to the last line except for the timestamp.

The disk space lists as such:
Disk hda7 (/) 40 %
Disk hda1 (/boot) 10 %
Disk hda8 (/home) 17 %
Disk hda5 (/tmp) 4 %
Disk hda2 (/usr) 59 %
Disk hda3 (/var) 11 %

None are even near full. I am at a loss. Any help would be appreciated. Thanks.
 

trparky

Well-Known Member
Apr 23, 2003
184
1
168
Anything from /var/messages and /usr/local/apache/logs/error_log?
 

Go3team

Member
Dec 4, 2003
12
0
151
trparky:
/var/log/messages shows nothing of consequence during the time listed below. The errors above came from /usr/local/apache/logs/error_log.

Renjith Rajan:
------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 655360 nobody 600 1
0x0052e2c1 12255233 postgres 600 17
0x0052e2c2 12288002 postgres 600 17
0x0052e2c3 12320771 postgres 600 17
0x00000000 884740 nobody 600 1
0x00000000 917509 nobody 600 1
0x00000000 1015814 nobody 600 1
0x00000000 1048583 nobody 600 1
0x00000000 1114120 nobody 600 1
0x00000000 1146889 nobody 600 1
0x00000000 1179658 nobody 600 1
0x00000000 1245195 nobody 600 1
0x00000000 1310732 nobody 600 1
0x00000000 1376269 nobody 600 1
0x00000000 1409038 nobody 600 1
0x00000000 1441807 nobody 600 1
0x00000000 1474576 nobody 600 1
0x00000000 1540113 nobody 600 1
0x00000000 1638418 nobody 600 1
0x00000000 1671187 nobody 600 1
0x00000000 1736724 nobody 600 1
0x00000000 1998869 nobody 600 1
0x00000000 2097174 nobody 600 1
0x00000000 2162711 nobody 600 1
0x00000000 2424856 nobody 600 1
0x00000000 2457625 nobody 600 1
0x00000000 2490394 nobody 600 1
0x00000000 2588699 nobody 600 1
0x00000000 2621468 nobody 600 1
0x00000000 2949149 nobody 600 1
0x00000000 2981918 nobody 600 1
0x00000000 3047455 nobody 600 1
0x00000000 3080224 nobody 600 1
0x00000000 3145761 nobody 600 1
0x00000000 3866658 nobody 600 1
0x00000000 3899427 nobody 600 1
0x00000000 4063268 nobody 600 1
0x00000000 4227109 nobody 600 1
0x00000000 4259878 nobody 600 1
0x00000000 4358183 nobody 600 1
0x00000000 4390952 nobody 600 1
0x00000000 4423721 nobody 600 1
0x00000000 4620330 nobody 600 1
0x00000000 4784171 nobody 600 1
0x00000000 5144620 nobody 600 1
0x00000000 5177389 nobody 600 1
0x00000000 5341230 nobody 600 1
0x00000000 5668911 nobody 600 1
0x00000000 5734448 nobody 600 1
0x00000000 5767217 nobody 600 1
0x00000000 5799986 nobody 600 1
0x00000000 5832755 nobody 600 1
0x00000000 5865524 nobody 600 1
0x00000000 5898293 nobody 600 1
0x00000000 5931062 nobody 600 1
0x00000000 5996599 nobody 600 1
0x00000000 6094904 nobody 600 1
0x00000000 6127673 nobody 600 1
0x00000000 6225978 nobody 600 1
0x00000000 6324283 nobody 600 1
0x00000000 6488124 nobody 600 1
0x00000000 6520893 nobody 600 1
0x00000000 6651966 nobody 600 1
0x00000000 6783039 nobody 600 1
0x00000000 7110720 nobody 600 1
0x00000000 7307329 nobody 600 1
0x00000000 7340098 nobody 600 1
0x00000000 7405635 nobody 600 1
0x00000000 7438404 nobody 600 1
0x00000000 7471173 nobody 600 1
0x00000000 7700550 nobody 600 1
0x00000000 7766087 nobody 600 1
0x00000000 7864392 nobody 600 1
0x00000000 7897161 nobody 600 1
0x00000000 7962698 nobody 600 1
0x00000000 7995467 nobody 600 1
0x00000000 8028236 nobody 600 1
0x00000000 8061005 nobody 600 1
0x00000000 8093774 nobody 600 1
0x00000000 8126543 nobody 600 1
0x00000000 8192080 nobody 600 1
0x00000000 8257617 nobody 600 1
0x00000000 8290386 nobody 600 1
0x00000000 8323155 nobody 600 1
0x00000000 8388692 nobody 600 1
0x00000000 8454229 nobody 600 1
0x00000000 8519766 nobody 600 1
0x00000000 8650839 nobody 600 1
0x00000000 8683608 nobody 600 1
0x00000000 8716377 nobody 600 1
0x00000000 8814682 nobody 600 1
0x00000000 8912987 nobody 600 1
0x00000000 8945756 nobody 600 1
0x00000000 8978525 nobody 600 1
0x00000000 9044062 nobody 600 1
0x00000000 9076831 nobody 600 1
0x00000000 9142368 nobody 600 1
0x00000000 9338977 nobody 600 1
0x00000000 9633890 nobody 600 1
0x00000000 9666659 nobody 600 1
0x00000000 9764964 nobody 600 1
0x00000000 9830501 nobody 600 1
0x00000000 9896038 nobody 600 1
0x00000000 9928807 nobody 600 1
0x00000000 9961576 nobody 600 1
0x00000000 10059881 nobody 600 1
0x00000000 10289258 nobody 600 1
0x00000000 10354795 nobody 600 1
0x00000000 10420332 nobody 600 1
0x00000000 10485869 nobody 600 1
0x00000000 10518638 nobody 600 1
0x00000000 10616943 nobody 600 1
0x00000000 10879088 nobody 600 1
0x00000000 11010161 nobody 600 1
0x00000000 11042930 nobody 600 1
0x00000000 11075699 nobody 600 1
0x00000000 11141236 nobody 600 1
0x00000000 11174005 nobody 600 1
0x00000000 11239542 nobody 600 1
0x00000000 11305079 nobody 600 1
0x00000000 11337848 nobody 600 1
0x00000000 11468921 nobody 600 1
0x00000000 11534458 nobody 600 1
0x00000000 11567227 nobody 600 1
0x00000000 11632764 nobody 600 1
0x00000000 11665533 nobody 600 1
0x00000000 11731070 nobody 600 1
0x00000000 12025983 nobody 600 1
 

chirpy

Well-Known Member
Verifed Vendor
Jun 15, 2002
13,466
31
473
Go on, have a guess
Have you checked the sizes of the apache log files in /etc/httpd/logs/ and /etc/httpd/domlogs, if any are near or over 2GB apache will have problems and you should rotate them out of the way.
 

Go3team

Member
Dec 4, 2003
12
0
151
Files in /etc/httpd/logs/:

access_log 200.9M
error_log 14.9M
httpd.pid 6
ssl_engine_log 3.7M
and a bunch of ssl_mutex.xxxxx files with 0 file length. Can these be deleted?

The largest file in /etc/httpd/domlogs is just about 21mb.
 

Go3team

Member
Dec 4, 2003
12
0
151
Is there a way to make the automatic restart more powerful? When I restart httpd under root, it does it right away. If I try in WHM it won't restart it either.
 

gemby

Well-Known Member
PartnerNOC
Feb 16, 2002
182
0
316
Pula, Croatia
cPanel Access Level
DataCenter Provider
I had same problems few weeks ago, i have no mod_python, but i think that this problem makes httpd itself due to unclean shutdowns, child crashings etc. Those days i had tons of ..reading.. requests.....

If restarting httpd does not clean at least one semaphore place, it will say segmet: no space left on device regarles that it has a lot of space on all partitions....

I found some "cleaning script" on net somewhere, an it solved my problems. Ok maybe not solved, but now at least my httpd-s are working now :))
__________________________________________________________
#!/bin/bash

ipcs -s | grep nobody | perl -e 'while (<STDIN>) {
@a=split(/\s+/); print `ipcrm sem $a[1]`}'

/scripts/restartsrv httpd
___________________________________________________________


I run it every day @ 2 am on my most affected box, and have it ready on all boxes, just in case.
 
Last edited:

rpmws

Well-Known Member
Aug 14, 2001
1,822
9
318
back woods of NC, USA
gemby said:
I had same problems few weeks ago, i have no mod_python, but i think that this problem makes httpd itself due to unclean shutdowns, child crashings etc. Those days i had tons of ..reading.. requests.....

If restarting httpd does not clean at least one semaphore place, it will say segmet: no space left on device regarles that it has a lot of space on all partitions....

I found some "cleaning script" on net somewhere, an it solved my problems. Ok maybe not solved, but now at least my httpd-s are working now :))
__________________________________________________________
#!/bin/bash

ipcs -s | grep nobody | perl -e 'while (<STDIN>) {
@a=split(/\s+/); print `ipcrm sem $a[1]`}'

/scripts/restartsrv httpd
___________________________________________________________


I run it every day @ 2 am on my most affected box, and have it ready on all boxes, just in case.

damn!!! I am glad I found this tonight!!! https would not run no matter what ..space looked fine everywhere. I had a DBD perl module fail at upcp tonight and https failed after that ..could get httpd to run but not https. Thanks for this post!! and the script!!!
 

gemby

Well-Known Member
PartnerNOC
Feb 16, 2002
182
0
316
Pula, Croatia
cPanel Access Level
DataCenter Provider
rpmws said:
damn!!! I am glad I found this tonight!!! https would not run no matter what ..space looked fine everywhere. I had a DBD perl module fail at upcp tonight and https failed after that ..could get httpd to run but not https. Thanks for this post!! and the script!!!
I am glad that i could help you, i have few dozens of such SOS scripts on servers, made to help me solve such small hedaches....