RHEL 4 and Account Creation / Transfer / Upgrade / Termination crashes servers

[elenlace]

I have had this issue now on and off for the last 3 months, on very seldom occassions, my RHEL 4 servers crash during account creation / transfer / upgrade / termination apparently anything related to quotas.

I have seen I'm not alone on this issue, several users have reported it. I was wondering what would happen if I turn quotas off, can I still control users space? It would be a temporary measure until Red Hat fixes the bug.

What do you think?

 

[elenlace]

Anyone? I can't believe this is only happenning to me.

This guy has the same problem: http://forums.cpanel.net/showthread.php?t=61367

Anyone from Cpanel care to comment?

 

[chirpy]

If you want help from cPanel you need to log a ticket with them through your license provider.

If you turn off linux disk quotas then there'd be no check at all on the amount of space clients can use.

 

[zigzam]

Its a RedHat Kernel bug.

 

[MN-Robert]

http://bugzilla.cpanel.net/show_bug.cgi?id=5106

Please vote for this bug, this kernel bug has been around for atleast 3 months redhat is ****.

 

[budway]

How many harddrives you have?

If more than one share partition style please (from console df -h).

 

[elenlace]

Hi budyway,

This is the result from df -h

Filesystem Size Used Avail Use% Mounted on
/dev/sda5 65G 49G 13G 80% /
/dev/sda1 99M 17M 77M 18% /boot
/dev/sda3 1012M 175M 786M 19% /tmp
none 1013M 0 1013M 0% /dev/shm
/dev/sdb1 68G 53G 12G 83% /backup
/tmp 1012M 175M 786M 19% /var/tmp

 

[budway]

Hi Elenlace,

First:
Have you tested the drives?

to run background test:
smartctl -t long /dev/sda

to view:
smartctl -a /dev/sda

Secound:
What kernel and DC (If TP what server/dc of TP)?

Last question when your server goes down does it still respond to ping and is a fsck required (does the server come back fast 1-5 minutes after the reboot).

 

[budway]

Hi Elenlace,

Also have you noted if your server only start's to crash when it begins to use swap (early swap usade like about 10MB of swap)?

 

[MN-Robert]

Dude this is a kernel bug that has been confirmed. Our servers have 8gb ram and sit at around 15% usage so its not a swap issue.

 

[MN-Robert]

New kernel is out, however theirs no details on whether the quota issue is fixed:

https://rhn.redhat.com/errata/RHSA-2007-0014.html

 

[budway]

This issue afect's only RHE 4.0 our RHE 3.0 also?

 

[elenlace]

Hi Elenlace,

First:
Have you tested the drives?

to run background test:
smartctl -t long /dev/sda

to view:
smartctl -a /dev/sda

Secound:
What kernel and DC (If TP what server/dc of TP)?

Last question when your server goes down does it still respond to ping and is a fsck required (does the server come back fast 1-5 minutes after the reboot).

Believe me is not related to the drives, one of my servers had this problem after just 2 days, it actually happenned after a transfer of accounts from another server. And yes the drives have been tested by techs at TP.

It has happenned both on servers in EV1's DC1 and DC2 so I don't see a correlation there. Is definately a Red Hat Kernel Bug and I don't see it has been fixed on the latest 42.0.8 kernel just out a few days ago.

Maybe Cpanel and the DCs can put some pressure on Red Hat to fix this as it is definately affecting several of us. It happens and when your server goes down it happened to me me once that it required a FSCK and it wouldn't boot again, so I had to restore the server using backups, I had around 20 hours of downtime on a weekday, believe me not fun at all.

This is a huge issue.

 

[elenlace]

This issue afect's only RHE 4.0 our RHE 3.0 also?

I believe is RHEL 4 only. I have 3 servers with RHEL 4 and 8 servers with RHEL 3 and I'm waiting for this issue to be fixed before I continue a migration to RHEL 4.

 

[elenlace]

Hi Elenlace,

Also have you noted if your server only start's to crash when it begins to use swap (early swap usade like about 10MB of swap)?

Nop, it happens when you create a new account, upgrade an account, terminate an account only, and of course during transfers as it calls the create a new account process.

Everything related to changing or creating quotas for a user seems to affect this.

If it continues to go on for several weeks without fix, I might consider disabling quotas for a while and just keep an eye on my users.

 

[MN-Robert]

We are testing the new kernel, I'll let you know how it goes.