Server crashes.

[moogle]

I searched.. but found nothing. Sorry if I missed a similar thread.



I have a server, Dual P3, and when I upgraded the server to Cpanel 7 it started crashing, regularly. I downgraded it to Cpanel 6 until yesterday, when I tried upgrading it again. While on Cpanel 6 there were no problems whatsoever. Now that I upgraded it back to 7, it's started crashing again.

I had my datacente rput in a ticket with Cpanel, they said everything was fine and that the server was wimpy. I have a Pentium 3 (single CPU) running Cpanel 7 and it does not have this problem.

While I wait for the datacenter to get in parts for an upgrade (which I'm not even sure will fix it) I need a solution. I'm going to paste the results of 'top' below, I see that swap completely deminishes and the load skyrockets, as well as an insane number of processes. I'm convinced thse are Cpanel processes (because of the upgrade) but I don't understand why the server can't handle 7. I'm still working on why the server doesn't have all the extra RAM I purchased (an extra 1G after the 512 it came with), and I understand that could effect this also..

------
7:42am up 11:53, 1 user, load average: 313.37, 318.39, 296.34
428 processes: 418 sleeping, 8 running, 0 zombie, 2 stopped
CPU0 states: 1.2% user, 12.8% system, 0.0% nice, 86.6% idle
CPU1 states: 0.11% user, 25.14% system, 0.1% nice, 73.10% idle
Mem: 902540K av, 893240K used, 9300K free, 0K shrd, 1988K buff
Swap: 1052216K av, 1052216K used, 0K free 9456K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
6 root 14 0 0 0 0 SW 12.0 0.0 22:05 kscand
5 root 11 0 0 0 0 DW 11.5 0.0 11:22 kswapd
32351 nobody 9 0 12524 492 116 D 1.1 0.0 0:01 httpd
1744 root 9 0 276 200 60 D 0.6 0.0 0:10 antirelayd
22381 root 10 0 504 160 104 S 0.6 0.0 0:06 sshd
2868 named 9 0 9604 8712 68 S 0.5 0.9 0:03 named
1493 root 9 0 152 136 52 D 0.4 0.0 0:06 syslogd
32163 rumbles 9 0 404 404 136 D 0.4 0.0 0:02 evolve.cgi
32208 nobody 9 0 12648 688 180 D 0.4 0.0 0:01 httpd
32543 mailman 9 0 600 600 120 D 0.4 0.0 0:00 python
32692 root 9 0 564 556 328 D 0.4 0.0 0:00 sendmail
32700 root 9 0 2196 1224 524 D 0.4 0.1 0:00 cppop
32758 mailnull 9 0 1220 1104 684 D 0.4 0.1 0:00 exim
309 root 13 0 408 408 284 D 0.4 0.0 0:00 sendmail
22480 root 10 0 696 648 204 R 0.3 0.0 1:35 top
32617 root 9 0 1096 1096 80 D 0.3 0.1 0:00 dcpumon



------
1:44pm up 5:55, 1 user, load average: 96.33, 41.89, 18.28
317 processes: 314 sleeping, 2 running, 1 zombie, 0 stopped
CPU0 states: 0.9% user, 16.6% system, 0.0% nice, 83.19% idle
CPU1 states: 0.13% user, 15.20% system, 0.0% nice, 84.1% idle
Mem: 902540K av, 895384K used, 7156K free, 0K shrd, 1056K buff
Swap: 1052216K av, 1052216K used, 0K free 5972K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
6 root 14 0 0 0 0 SW 15.9 0.0 6:34 kscand
5 root 15 0 0 0 0 DW 7.7 0.0 1:26 kswapd
2891 nobody 9 0 99716 37M 448 S 0.5 4.2 0:32 httpd
15266 fingerpr 9 0 2180 1128 304 D 0.4 0.1 0:00 cppop
15320 root 10 0 1788 376 120 D 0.4 0.0 0:00 cppop
1783 root 19 19 5240 2592 240 D N 0.3 0.2 0:48 cpanellogd
14979 nobody 9 0 13800 2040 752 D 0.4 0.2 0:00 httpd
15234 nobody 9 0 12920 860 448 D 0.4 0.0 0:00 httpd
15259 mailman 9 0 676 676 288 D 0.4 0.0 0:00 python
15268 mysql 10 0 16184 7816 92 D 0.4 0.8 0:00 mysqld
15309 root 9 0 1816 388 152 D 0.4 0.0 0:00 cppop
15314 mailman 9 0 480 472 364 D 0.4 0.0 0:00 crond
148 root 9 0 0 0 0 SW 0.3 0.0 0:07 kjournald
2802 nobody 9 0 81736 21M 420 D 0.3 2.4 0:28 httpd
14659 nobody 9 0 17424 5556 488 D 0.3 0.6 0:00 httpd
15235 root 9 0 144 144 76 D 0.3 0.0 0:00 suexec

cPanel.net Support Ticket Number:

[ssilvius]

I have zero idea whats going on as there is no where near enough data to even start to guess, but I add a new box when I see my load to 2.0.

400 processes? how many accounts you got on this thing?

You're not hosting a bunch of tomcat sites are you?

cPanel.net Support Ticket Number:

[moogle]

This server never went up in processes like that. It's usually around 100, and it never did that with CPanel 6. This server is over a year old, if it were a problem I caused something would have happened long before the upgrade to Cpanel7.

cPanel.net Support Ticket Number:

[ssilvius]

Is the sendmail in the first one a symlink for exim or do you have both installed?

I just had a client let their password/username get into the wild and some wonderful chinese hackers/spammers got ahold of it and installed sendmail into the jailshell and started relaying.... or so the story from the client goes... the access logs do show real chinese IP addresses from alot of HK ISPs

[moogle]

I believe it's a symlink, it says 'sendmail@'.
I know this is related to Cpanel 7, I downgraded and it stopped, then it started again when I upgraded. And I do know people have gotten into this machine, which is why I needed to upgrade it, for security. But I've had any hacks or trojans cleaned out.

cPanel.net Support Ticket Number:

[ssilvius]

Not to be a prick or anything, but security does not come from cpanel. That's what informed and knowledgeable admins are for. If I even have the smallest hint of something fishy, accounts are transfered to other machines, I grab the logs and compare to the syslog database to see whats going on while the machine is getting re-imaged, hardened and all clients that where on that machine forced into password changes. anal? yes. but downtime because of the mystery load is not acceptable to myself, my co-workers nor our clients. Once you're rooted there is no cleaning things out and going along unless you have the know-how to do so.

I suggest a rebuild from the ground up and before a single account is added install tripwire, know what changes and why it changes. Troubleshooting something like this from afar without having everything in front of someone to troubleshoot with is damn near impossable.

cPanel.net Support Ticket Number:

[moogle]

Regardless, this machine needs to be fixed and starting it from scratch right now is not possible. If you can't (or won't) help, that's fine. I really don't have time for the shoulda coulda's, I need to stop this from happening while I wait.

The Cpanel version does offer a certain level of security, just as the version of Apache does. People learn how to get inside older scripts, which is why we upgrade to versions that have those certain things fixed. When people have gotten into the machine (greymatter and phpshell style) I have found it, cleared it out, and proper binaries have been reinstalled if needed, etc.

Right now what I need is the reason for this, and I need a little help fixing it, not you running me down for possibly not being as advanced in the past.

If anyone needs any more information from me to try to help, please ask. And thanks for anyone who is willing to help.

cPanel.net Support Ticket Number: