No space left on device (Apache error) - but there was

[jeroman8]

Hi!

Anyone know why this happens ?

Apache is down due to "No space left on device".
It say so in apache error log.

But listed i WHM there is more than 1GB left on /usr and no
other partion is full.
I even removed log files and other stuff so 1GB more was free.

/scripts/restartsrv_httpd and service httpd restart does not work.
Error still say no space left on device.

After a reboot it works OK.

Why did I have to reboot the machine so apache (or system)
would see that it's not full ?

Just wanna know for the future.

Thanks!

[sampride]

Run this as root:

ipcs -s | grep nobody | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}'

It will fix this error.

[rhenderson]

Anyone know why this happens ?

Apache is down due to "No space left on device".
It say so in apache error log.

But listed i WHM there is more than 1GB left on /usr and no
other partion is full.
I even removed log files and other stuff so 1GB more was free.

/scripts/restartsrv_httpd and service httpd restart does not work.
Error still say no space left on device.

After a reboot it works OK.

Why did I have to reboot the machine so apache (or system)
would see that it's not full ?

Just wanna know for the future.

Thanks!

It happens because of too many semaphores, run
# ipcs -s

You should get the following:

------ Semaphore Arrays --------
key semid owner perms nsems
0x00000000 12779528 nobody 600 1
0x00000000 12812297 nobody 600 1
0x00000000 12845066 nobody 600 1
0x00000000 12877835 nobody 600 1
0x00000000 12910604 nobody 600 1
0x00000000 12943373 nobody 600 1
0x00000000 12976142 nobody 600 1
0x00000000 13008911 nobody 600 1
0x00000000 13041680 nobody 600 1

But I bet you get a lot more and the owners are your clients.

Run this as root:

ipcs -s | grep nobody | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}'

It will fix this error.

That might clear it up enough for a restart but it will not clear the semaphores owned by others than nobody, since your just searching for nobody. Do that wait a few hours and try to restart apache and it will fail with your error again. So it is not a fix, more like a bandaid or an emergency restart.

We ran into this and it was very frustrating for us to track down. We found that when you build apache if you go to the exhaustive list and have Mod FCGID - fastcgi support and Fastcgi (This option will make the following changes to your profile prior to the build) Enables: Mod FCGID, it will create a lot more semaphores with it enabled, it will create semaphores for anyone using crons in a certain way, then if you have anything that restarts apache it will fail with your error.

Rebuilding apache and taking that option out corrected our problem, which is identical to your problem.
Good Luck!!