Is CentOS 4.3 , kernel 2.6.9-34 stable ?

[fastdns]

Hi

I am using :

OS : CentOS release 4.3 (Final)

CPanel Version : WHM 10.8.0 cPanel 10.8.1-R113

Kernel Version : 2.6.9-34.ELsmp

I am observing excessive server load. Usually it runs at 20-30 cpu and spikes up to 65 cpu when I have to stop exim to bring it down.

I have been complaining about high server loads for the past 2 months and everytime I was told that there is nothing wrong with the version of the kernel that I have.

Now, other people using Cent Os 4.3 have reported similar issues with high server load and memory management.

Please read this thread : http://forums.cpanel.net/showthread.php?t=54280&highlight=centos+high+server+load

Read the bug report at : http://bugs.centos.org/view.php?id=1174

have thoroughlly checked my box and there is nothing "unusual" happening on my server that should cause such high server loads, almost to the tune of 70 CPU ??? All that I can see if that the server load goes high while waiting for an i/o read from HD.

My current top shows :

top - 23:50:03 up 8 days, 2:55, 2 users, load average: 65.59, 45.05, 27.05
Tasks: 388 total, 2 running, 374 sleeping, 0 stopped, 12 zombie
Cpu(s): 5.6% us, 3.9% sy, 0.0% ni, 0.0% id, 90.2% wa, 0.3% hi, 0.0% si
Mem: 969596k total, 300172k used, 669424k free, 3312k buffers
Swap: 4192956k total, 333064k used, 3859892k free, 16540k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12125 mailnull 17 0 8820 3316 2300 S 1.2 0.3 0:00.04 exim
38 root 15 0 0 0 0 D 0.9 0.0 19:15.24 kswapd0
18514 root 15 0 3032 1028 676 S 0.6 0.1 0:16.75 top
4123 root 16 0 3888 1036 680 R 0.6 0.1 0:06.51 top
12161 mailnull 16 0 8816 3336 2324 D 0.6 0.3 0:00.07 exim
12162 cpanel 18 0 9940 3712 1952 D 0.6 0.4 0:00.03 php
12169 root 18 0 11864 3736 1368 D 0.6 0.4 0:00.02 cpsrvd
199 root 15 0 0 0 0 D 0.3 0.0 4:45.77 kjournald
1949 autobulb 15 0 7428 3652 3096 D 0.3 0.4 0:00.04 boxtrapper
8149 nobody 15 0 17796 5548 1400 S 0.3 0.6 0:00.07 httpd
12166 root 16 0 8236 3716 2684 S 0.3 0.4 0:00.03 exim
12182 mailnull 16 0 8808 3124 2168 S 0.3 0.3 0:00.03 exim

Any help on this from anyone in this forum would be highly appreciated.

Regards

 

[AndyReed]

Kernel Version : 2.6.9-34.ELsmp

I am observing excessive server load. Usually it runs at 20-30 cpu and spikes up to 65 cpu when I have to stop exim to bring it down.

The kernel is not the only cause of high server load on a CentOS server. Down/upgrading might help, in your case you need to go after the source of the trouble. I suggest you search these forums since this issue has been discussed so many times. Good luck!

 

[fastdns]

Dear Andy

I have spent so much time in this forum searcing for something that would help me, but all in vain.

Problem is that the same number of accounts were running on Fedora 4 before I shifted my server to this new datacenter and got a CentOS. My new server is running with even lesser accounts now, I have shut down services like spamassassin also, I have moved heavy email users to other servers, but I fail to understand that even then , why is is always running at a high usage.

For example, see this top :

top - 02:25:30 up 7 days, 5:30, 2 users, load average: 20.25, 26.61, 23.83
Tasks: 248 total, 2 running, 238 sleeping, 1 stopped, 7 zombie
Cpu(s): 5.0% us, 7.3% sy, 0.0% ni, 0.0% id, 87.4% wa, 0.3% hi, 0.0% si
Mem: 969596k total, 223720k used, 745876k free, 1052k buffers
Swap: 4192956k total, 323824k used, 3869132k free, 15320k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
38 root 15 0 0 0 0 D 6.0 0.0 17:27.63 kswapd0
12697 mailnull 16 0 9284 4372 3280 D 1.0 0.5 0:00.03 exim
12952 root 18 0 10068 3740 2692 S 1.0 0.4 0:00.06 exim
6435 root 16 0 3140 1052 680 S 0.6 0.1 0:06.31 top

The "D" state is the real issue in those processes. In the snippet above, I can see two exim processes, and a kswapd (kernel swap manager) are in D state, meaning they are waiting on feedback from hardware. These processes can get stuck in an infinite read() if the hardware never times out, such as reading from a harddrive.

I'm not sure how much a kernel update will help such a problem, however it's certainly worth a shot. But my datacenter team is refusing to do this as they say that the last stable version is -34 as mentioned on http://mirror.centos.org/centos/4.3/updates/i386/RPMS/

I am stuck....thinking of moving over back to the old company

 

[AndyReed]

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12697 mailnull 16 0 9284 4372 3280 D 1.0 0.5 0:00.03 exim
12952 root 18 0 10068 3740 2692 S 1.0 0.4 0:00.06 exim

a shot. But my datacenter team is refusing to do this as they say that the last stable version is -34 as mentioned on http://mirror.centos.org/centos/4.3/updates/i386/RPMS/

kernel-2.6.9-40 is the latest and most stable version. We upgraded our clients' kernel to this release and works just fine. If you are interested in learning more about high server load, read this thread: http://forums.cpanel.net/showthread.php?t=50421 Hope this helps!

 

[MMarko]

It is possible to compile latest kernel 2.6.17.3 from source on production server without any problems.


Tutorial - http://webhostingtalk.com/showthread.php?t=489019&highlight=update+kernel+on+centos+4



Default CentOS 4 kernel 2.6.9-22 has a lots of problems and bugs with I/O and swapping...

 

[rjf1505]

Hi

I am using :

OS : CentOS release 4.3 (Final)

CPanel Version : WHM 10.8.0 cPanel 10.8.1-R113

Kernel Version : 2.6.9-34.ELsmp

I am observing excessive server load. Usually it runs at 20-30 cpu and spikes up to 65 cpu when I have to stop exim to bring it down.

I have been complaining about high server loads for the past 2 months and everytime I was told that there is nothing wrong with the version of the kernel that I have.

Now, other people using Cent Os 4.3 have reported similar issues with high server load and memory management.


Regards

I have excactly the same problem !

My host has also swapped my hardware (PERFECT SERVICE !) and the problem stays.
When disabling EXIM i can hold the server under control. I only start exim once in a while.

The server worked fine until several weeks ago, i am going to check if i can see when the kernel was updated.

Is the solution up-down grading the kernel ?

Thanks

 

[bidouilleur]

I upgraded several servers with load problems (all Centos) to 2.6.9-39.0.2.EL


and they are now all calm and nice again.

 

[AndyReed]

It is possible to compile latest kernel 2.6.17.3 from source on production server without any problems.

Yes, it is possible, but it depends on the level of expertise with Linux/FreeBSD OS. If you compile Kernel from source, and just in case it didn't work, you can always switch back to the previous working version you had.

 

[gupi]

Well, http://people.redhat.com/~jbaron/rhel4/RPMS.kernel/ contains now a 2.6.9-42 kernel.
Having the same issue with 2.6.9-34 kernel, i've installed it and rebooted.
All went smooth and worked perfect (please note that is a personal oppinion, not an advice).

 

[trhosting.net]

The original source file of this kernel is:
ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/4ES/en/os/SRPMS/kernel-2.6.9-42.EL.src.rpm

You can compile this kernel or use the kernel that gupi wrote.

 

[mctDarren]

I upgraded several servers with load problems (all Centos) to 2.6.9-39.0.2.EL


and they are now all calm and nice again.

Same here, going to this kernel fixed my woes for a couple boxes.

 

[chilihost]

I have upgraded a couple of my servers to 2.6.9-39.0.2.ELsmp and it is WAY WAY WAY better...see this thread: http://forums.cpanel.net/showthread.php?t=56045

The upgrade is very easy too, you just download the RMPs from the testing area on centOS. I have developed the following set of instructions if you are using grub as your os loader:

cd /var/spool/up2date

wget http://dev.centos.org/centos/4/testing/i386/RPMS/kernel-2.6.9-39.0.2.EL.i686.rpm
wget http://dev.centos.org/centos/4/testing/i386/RPMS/kernel-devel-2.6.9-39.0.2.EL.i686.rpm
wget http://dev.centos.org/centos/4/testing/i386/RPMS/kernel-hugemem-devel-2.6.9-39.0.2.EL.i686.rpm
wget http://dev.centos.org/centos/4/testing/i386/RPMS/kernel-smp-2.6.9-39.0.2.EL.i686.rpm
wget http://dev.centos.org/centos/4/testing/i386/RPMS/kernel-smp-devel-2.6.9-39.0.2.EL.i686.rpm

rpm -ivh kernel-2.6.9-39.0.2.EL.i686.rpm kernel-devel-2.6.9-39.0.2.EL.i686.rpm kernel-hugemem-devel-2.6.9-39.0.2.EL.i686.rpm kernel-smp-2.6.9-39.0.2.EL.i686.rpm kernel-smp-devel-2.6.9-39.0.2.EL.i686.rpm

Then to be safe, set your server to boot into the new kernel only once:

vi /boot/grub/grub.conf

in grub, change order of new kernel to second, make the old kernel the first one. Then do the following:

grub

wait a few seconds for this to come up, then type the following lines

savedefault --default=1 --once
quit

then reboot your server and check to see if the new kernel was successful:
uname -r

if it is then edit grub.conf so that the new kernel is the first in the list. If this does not work, simply reboot / power cycle and you should come up with your old kernel.

Good luck, use these instructions and this test kernel AT YOUR OWN RISK!!!

 

[trhosting.net]

Redhat released the official kernel-2.6.9-42.
This is the latest stable kernel for RHEL4 (and CentOS 4.3).

We have compiled and installed this kernel on one of our servers.
Now we are testing the performance.
I will notify you about the performance.

If anyone interests on compiling this kernel, see below:

wget ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/4ES/en/os/SRPMS/kernel-2.6.9-42.EL.src.rpm
rpm -ihv kernel-2.6.9-42.EL.src.rpm

The sources of the kernel is now in /usr/src/redhat

cd /usr/src/redhat
rpmbuild -bb --target=i686 SPECS/kernel-2.6.spec

It takes about 30 minutes to compile the kernel. After a successfull compile, the new kernel rpms are located on "/usr/src/redhat/RPMS/i686" folder.

-rw-r--r--  1 root root  11327798 Aug 14 05:27 kernel-2.6.9-42.EL.i686.rpm
-rw-r--r--  1 root root 316150714 Aug 14 05:53 kernel-debuginfo-2.6.9-42.EL.i686.rpm
-rw-r--r--  1 root root   3880571 Aug 14 05:27 kernel-devel-2.6.9-42.EL.i686.rpm
-rw-r--r--  1 root root  10818123 Aug 14 05:29 kernel-hugemem-2.6.9-42.EL.i686.rpm
-rw-r--r--  1 root root   3906847 Aug 14 05:29 kernel-hugemem-devel-2.6.9-42.EL.i686.rpm
-rw-r--r--  1 root root  10939279 Aug 14 05:28 kernel-smp-2.6.9-42.EL.i686.rpm
-rw-r--r--  1 root root   3897131 Aug 14 05:28 kernel-smp-devel-2.6.9-42.EL.i686.rpm

You can now install what kernel you want.

 

[trhosting.net]

UPDATE:

Now it is peak time for the server that we upgraded and it seems working perfectly.

 

[fastdns]

Redhat released the official kernel-2.6.9-42.
This is the latest stable kernel for RHEL4 (and CentOS 4.3).

Where can I get the latest official kernel ? the link that you posted does not work :

ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/4ES/en/os/SRPMS/kernel-2.6.9-42.EL.src.rpm

regards,

Dhiraj

 

[trhosting.net]

You can download it from:
ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/4ES/en/os/SRPMS/

If you want you can download the packages that i compiled from http://www.trhosting.net/kernel/

These packages are for Centos 4.x i686

 

[chirpy]

No need now, since CentOS have released the 2.6.9-42.0.2 kernel for v4 now. You can get it from your freindly yum mirror

 

[MMarko]

Does anyone know why cpanel service fails to start when centOS is booted with kernel 2.4.33?

thanks

 

[minzhong]

Hi all.

I'm currently running Dual Xeons.

Do dual processor systems require different 'type' of kernel?

I tried other kernels in the past but they don't work.

I'm currently using the default 2.6.9-34.ELsmp from the default CentOS 4.3 installation.

Please advise!

Thank you..

 

[chirpy]

They simply need to smp kernel. You should be able to use the latest CentOSv4 kernel unless you have RAID hardware or the like that the manufacturer hasn't bothered supporting (there are a few). The kernel you are running (2.6.9-34.ELsmp) is now old and vulnerable to root escalation compromises.