No space left on device (Apache error) - but there was

jeroman8

Well-Known Member
Mar 14, 2003
410
0
166
Hi!

Anyone know why this happens ?

Apache is down due to "No space left on device".
It say so in apache error log.

But listed i WHM there is more than 1GB left on /usr and no
other partion is full.
I even removed log files and other stuff so 1GB more was free.

/scripts/restartsrv_httpd and service httpd restart does not work.
Error still say no space left on device.

After a reboot it works OK.

Why did I have to reboot the machine so apache (or system)
would see that it's not full ?

Just wanna know for the future.

Thanks!
 

sampride

Active Member
Jul 8, 2005
25
0
151
Run this as root:

ipcs -s | grep nobody | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}'

It will fix this error.
 

rhenderson

Well-Known Member
Apr 21, 2005
783
2
168
Oklahoma
cPanel Access Level
Root Administrator
Anyone know why this happens ?

Apache is down due to "No space left on device".
It say so in apache error log.

But listed i WHM there is more than 1GB left on /usr and no
other partion is full.
I even removed log files and other stuff so 1GB more was free.

/scripts/restartsrv_httpd and service httpd restart does not work.
Error still say no space left on device.

After a reboot it works OK.

Why did I have to reboot the machine so apache (or system)
would see that it's not full ?

Just wanna know for the future.

Thanks!
It happens because of too many semaphores, run
# ipcs -s

You should get the following:

------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 12779528 nobody 600 1
0x00000000 12812297 nobody 600 1
0x00000000 12845066 nobody 600 1
0x00000000 12877835 nobody 600 1
0x00000000 12910604 nobody 600 1
0x00000000 12943373 nobody 600 1
0x00000000 12976142 nobody 600 1
0x00000000 13008911 nobody 600 1
0x00000000 13041680 nobody 600 1

But I bet you get a lot more and the owners are your clients.


Run this as root:

ipcs -s | grep nobody | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}'

It will fix this error.
That might clear it up enough for a restart but it will not clear the semaphores owned by others than nobody, since your just searching for nobody. Do that wait a few hours and try to restart apache and it will fail with your error again. So it is not a fix, more like a bandaid or an emergency restart.

We ran into this and it was very frustrating for us to track down. We found that when you build apache if you go to the exhaustive list and have Mod FCGID - fastcgi support and Fastcgi (This option will make the following changes to your profile prior to the build) Enables: Mod FCGID, it will create a lot more semaphores with it enabled, it will create semaphores for anyone using crons in a certain way, then if you have anything that restarts apache it will fail with your error.

Rebuilding apache and taking that option out corrected our problem, which is identical to your problem.
Good Luck!!
 
Last edited: