were off topic now.. but, since june of 2001 till today nobody should have had a uptime of more than 120 days... Check out errata your self... most of the time theirs a new kernel out twice every 3 months...
It should also depend on what security risks are involved in each patch. At times a box with one site on it, where you know what being run may not have a issue and can be left alone for sake of no downtime.
BEWARE!
We added the code to the file and before the server could be rebooted, it crashed. It needed a hard reboot at the NOC and came back with this error:
Error failed to mount dir or dir not found.
The NOC said the hard drive would only come back up in read-only.
12 hours later, we finally had the server back online after changing out the hard drive, re-installing cpanel and all of the accounts.
This is what our NOC had to say about the reason for the catastrophe:
"yes it was that code - Im not 100% sure but I think
that is dependent on the size and blocks of the hard drive - if you have the same size drive they did, it would probably work fine - they might have had an 80g or something like that - or a smaller drive with different block size"
Here is what a systems administrator had to say about the situation:
"We have access to several 80gb and 40gb servers, all have the same file-max limits (cannot be drive dependant or one would have crashed). Below is exactly what the file-max is for, it is kernel and ram related not harddrive related.
The file-max file /proc/sys/fs/file-max sets the maximum number of
file-handles that the Linux kernel will allocate. We generally tune this file to improve the number of open files by increasing the value of /proc/sys/fs/file-max to something reasonable like 256 for every 4M of RAM we have: i.e. for a machine with 128 MB of RAM, set it to 8192 - 128/4=32 32*256=8192.
The default setup for the file-max parameter under Red Hat Linux is: "4096" "
Here is a quote from one of the Unix Engineers that I asked about this issue:
"It's not harddrive related. Maximum number of file-handles has to do with memory, you want use it you set 256 for every 4MB or memory. In order to use 64000, you need 1GB of memory. Those applies to kernel 2.2x only.
for kernel 2.4x (redhat) this is best way you set:
edit /etc/sysctl.conf, and add the following line:
e.g. fs.file-max = 64000"
I'm not convinced at all that it was this change in the code that took the server down, but it is very odd that it happened just after the file was modified.
Can anybody (Sash?) shed any light on what happened to us?
cPanel.net Support Ticket Number:
We had a server that looked like it was starting into this cycle of going down for no apparent reason, so we tried this fix.
We added the code to the file and before the server could be rebooted, it crashed. It needed a hard reboot at the NOC and came back with this error:
Error failed to mount dir or dir not found.
The NOC said the hard drive would only come back up in read-only.
12 hours later, we finally had the server back online after changing out the hard drive, re-installing cpanel and all of the accounts.
This is what our NOC had to say about the reason for the catastrophe:
"yes it was that code - Im not 100% sure but I think
that is dependent on the size and blocks of the hard drive - if you have the same size drive they did, it would probably work fine - they might have had an 80g or something like that - or a smaller drive with different block size"
Here is what a systems administrator had to say about the situation:
"We have access to several 80gb and 40gb servers, all have the same file-max limits (cannot be drive dependant or one would have crashed). Below is exactly what the file-max is for, it is kernel and ram related not harddrive related.
The file-max file /proc/sys/fs/file-max sets the maximum number of
file-handles that the Linux kernel will allocate. We generally tune this file to improve the number of open files by increasing the value of /proc/sys/fs/file-max to something reasonable like 256 for every 4M of RAM we have: i.e. for a machine with 128 MB of RAM, set it to 8192 - 128/4=32 32*256=8192.
The default setup for the file-max parameter under Red Hat Linux is: "4096" "
Here is a quote from one of the Unix Engineers that I asked about this issue:
"It's not harddrive related. Maximum number of file-handles has to do with memory, you want use it you set 256 for every 4MB or memory. In order to use 64000, you need 1GB of memory. Those applies to kernel 2.2x only.
for kernel 2.4x (redhat) this is best way you set:
edit /etc/sysctl.conf, and add the following line:
e.g. fs.file-max = 64000"
I'm not convinced at all that it was this change in the code that took the server down, but it is very odd that it happened just after the file was modified.
Can anybody (Sash?) shed any light on what happened to us?
cPanel.net Support Ticket Number:
We've done this procedure on ~25 servers without any problems.
All our servers run either 36 or 73 SCSII drives, with RAID1 or RAID5, and have between 512 meg and 2 gig of DDR RAM.
I'd be interested in hearing other people's experiences.
RPMWS: How has the fixed worked for you?
Mike
cPanel.net Support Ticket Number:
All our servers run either 36 or 73 SCSII drives, with RAID1 or RAID5, and have between 512 meg and 2 gig of DDR RAM.
I'd be interested in hearing other people's experiences.
RPMWS: How has the fixed worked for you?
Mike
cPanel.net Support Ticket Number:
Just Install Cpanel On new server 3 day , First day it down , i check that and when i reboot found antirelayd down, failed , also update to new kernel , server load is ok but ram use is too high
you see >
total used free shared buffers cached
Mem: 1030200 975332 54868 0 174532 629740
-/+ buffers/cache: 171060 859140
Swap: 2096472 852 2095620
Total: 3126672 976184 2150488
My god just restart one day only , less then 5 customer in this server!
My new server is Celeron 1.8 + 1G DDR Ram and 60GB HD!
What can i do , never want server down!
cPanel.net Support Ticket Number:
you see >
total used free shared buffers cached
Mem: 1030200 975332 54868 0 174532 629740
-/+ buffers/cache: 171060 859140
Swap: 2096472 852 2095620
Total: 3126672 976184 2150488
My god just restart one day only , less then 5 customer in this server!
My new server is Celeron 1.8 + 1G DDR Ram and 60GB HD!
What can i do , never want server down!
cPanel.net Support Ticket Number:
running 25 plus days finally with this fix. also added the httpd restart in cron nightly.
cPanel.net Support Ticket Number:
cPanel.net Support Ticket Number:
If your system does crash, you can see what processes where eating up ram/cpu before it did by looking at /var/log/dcpumon/boot.[boot time here in unix time]/*.gz
cPanel.net Support Ticket Number:
cPanel.net Support Ticket Number:
Odd.
I implemented this fix. It worked well for about a week. Now for some reason all the settings have been reset. For instance, I type ulimit -u and it's no longer set to unlimited. The file /proc/sys/fs/file-max is set way too high. What would run after the /etc/rc.d/rc.local script that would reset these options? The server is running away again.
cPanel.net Support Ticket Number:
I implemented this fix. It worked well for about a week. Now for some reason all the settings have been reset. For instance, I type ulimit -u and it's no longer set to unlimited. The file /proc/sys/fs/file-max is set way too high. What would run after the /etc/rc.d/rc.local script that would reset these options? The server is running away again.
cPanel.net Support Ticket Number:
Recently we have this problem as well but only with one of our boxes, Linux 8 and latest version on CPanel.
cPanel.net Support Ticket Number:
cPanel.net Support Ticket Number:
Any news?
We have lots of :
filelimits: Increasing file system limits succeeded
on boot.log, no high process, no mem usage, just crash.
cPanel.net Support Ticket Number:
We have lots of :
filelimits: Increasing file system limits succeeded
on boot.log, no high process, no mem usage, just crash.
cPanel.net Support Ticket Number:
RPMUP2?????
Before trying the fixes suggested here I thought I would look at the processes that run before the crash.
In my case is RPMUP2.
This process starts at 5 am using 65% CPU and about 70% MEM. it goes on for 208 minutes and finally crashes the server at about 8 am with 99.9% CPU and 95% MEM
Can I disable rpmup2 or at least run it at desired intervals in /etc/cpupdate.conf
Is this a bug in rpmup?
Please help. My clients come to work at 8 am and they are yelling.
cPanel.net Support Ticket Number:
Before trying the fixes suggested here I thought I would look at the processes that run before the crash.
In my case is RPMUP2.
This process starts at 5 am using 65% CPU and about 70% MEM. it goes on for 208 minutes and finally crashes the server at about 8 am with 99.9% CPU and 95% MEM
Can I disable rpmup2 or at least run it at desired intervals in /etc/cpupdate.conf
Is this a bug in rpmup?
Please help. My clients come to work at 8 am and they are yelling.
cPanel.net Support Ticket Number:
As fmalekpour noted, my systems are also reverting back to their original settings.
I set the correct lines in rc.local, and upon rebooting the server, the limits (aside from file-max) are not set as they were.
I placed "fs.file-max 64000" in sysctl.conf now, just in case it decides to revert back, also. =)
But, for the others, I'm a bit sketchy on how they would (or could) be added to sysctl.conf.
One thing to note, though... when httpd is started, open files limit is reset, it looks like (/etc/rc.d/init.d/httpd)
This issue has plagued all of our servers since day 1.
Regards,
Matt
I set the correct lines in rc.local, and upon rebooting the server, the limits (aside from file-max) are not set as they were.
I placed "fs.file-max 64000" in sysctl.conf now, just in case it decides to revert back, also. =)
But, for the others, I'm a bit sketchy on how they would (or could) be added to sysctl.conf.
One thing to note, though... when httpd is started, open files limit is reset, it looks like (/etc/rc.d/init.d/httpd)
This issue has plagued all of our servers since day 1.
Regards,
Matt
I've been having this problem across all of my servers but two for the past 8 months. Loads are not abnormal and crashes are common leaving everything down except for being able to ping the server. Oddly enough, the two servers that were running ProFTPD had been running fine for about 4 months, only problems were memory-related and resolved quickly and crashes ceased. The rest of the 10 servers, however, were running Pure-FTPD, and crashed on a regular basis.
I have tried the fix noted in this thread, but the same day I completed the modification to a certain server, the settings went back to normal and the machine crashed.
When I'd check the logs after the server came back online, there would always be something relating to the FTP daemon within one or two lines of when the crash actually took place. There were also notes of these filesystem limit increases. There was never anything directly relating to why the server crashed in the messages log or within the top logs, everything looked normal.
On average, we have a server crash every couple of days. Sometimes they'll get mad at me (heh) and crash a couple times during a single day (as one server did this week). On all but two of my servers, I am now running ProFTPD, the build linked below from Nick. So far, so good.
http://ftp.cpanel.net/proftpd-1.2.9rc2tls-3_linuxprivs.i386.rpm
Anyone else notice anything similar? Or have I just had a bit of luck over the past 48 hours? heh
Oddly similar these crashes I've had are to the ones described in this thread. I can't say this -is- the fix, but it's worked for me so far. The fix described previously in this thread was overridden on all of my servers after a reboot. (tried in /etc/rc.d/rc.local and in /etc/sysctl.conf ... no luck making it stick)
Another note, the filesystem file totally fubars my one server running Redhat 9, every few hours. I had to rename the file to keep my server from being inaccessible.
Regards,
Matt
I have tried the fix noted in this thread, but the same day I completed the modification to a certain server, the settings went back to normal and the machine crashed.
When I'd check the logs after the server came back online, there would always be something relating to the FTP daemon within one or two lines of when the crash actually took place. There were also notes of these filesystem limit increases. There was never anything directly relating to why the server crashed in the messages log or within the top logs, everything looked normal.
On average, we have a server crash every couple of days. Sometimes they'll get mad at me (heh) and crash a couple times during a single day (as one server did this week). On all but two of my servers, I am now running ProFTPD, the build linked below from Nick. So far, so good.
http://ftp.cpanel.net/proftpd-1.2.9rc2tls-3_linuxprivs.i386.rpm
Anyone else notice anything similar? Or have I just had a bit of luck over the past 48 hours? heh
Oddly similar these crashes I've had are to the ones described in this thread. I can't say this -is- the fix, but it's worked for me so far. The fix described previously in this thread was overridden on all of my servers after a reboot. (tried in /etc/rc.d/rc.local and in /etc/sysctl.conf ... no luck making it stick)
Another note, the filesystem file totally fubars my one server running Redhat 9, every few hours. I had to rename the file to keep my server from being inaccessible.
Regards,
Matt
Last edited:
| Thread starter | Similar threads | Forum | Replies | Date |
|---|---|---|---|---|
| S | In Progress CPANEL-42630 - Determining best certificate logic | Server Management | 4 | |
| R | cPanel 500 Error ID 1db081a92f36e | Server Management | 1 | |
| R | Cpanel/whm not working | Server Management | 1 | |
| D | Cpanel / WHM Left Column Font Changed | Server Management | 5 | |
| G | Cpanel Server crashes | Server Management | 0 |