HUGE Load, Runaway server, what is happening!

scottgem

Member
Feb 24, 2003
10
0
151
I'm having a problem with one of my servers! It keeps running itself into the ground... I'm completely stumped. It's up for a matter of 40 minutes and the load spikes to 129.00+ ! If the httpd processes are not killed its gone and it must be manually rebooted at the NOC.

I just regained control of it by luckily getting a kill httpd command in... my result after it coming back to life is a load average: 241.50, 261.48, 183.51

What are the possibilities for this? The server is definately loaded but I don't think thats the true issue, take a look at the mem report below. It doesn't make much sense to me because it displays the load at 129.00 but cpu is 91.7% idle!? How is this possible? The server is fully updated, with RHN as well as C-Panel stable release. Newest apache core installed as well. Kernel is up to date (2.4.20-18.7)

-----------RESULTS OF TOP-----------------
12:07pm up 52 min, 2 users, load average: 129.00, 70.78, 39.22
257 processes: 230 sleeping, 25 running, 2 zombie, 0 stopped
CPU states: 2.9% user, 5.2% system, 0.0% nice, 91.7% idle
Mem: 514120K av, 507772K used, 6348K free, 0K shrd, 5068K buff
Swap: 2048276K av, 1111316K used, 936960K free 27292K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
12454 nobody 15 0 20596 15M 1672 R 0.7 3.1 0:00 httpd
12461 nobody 18 0 9904 5056 1068 R 0.7 0.9 0:00 httpd
6 root 12 0 0 0 0 SW 0.5 0.0 0:17 kscand
12821 nobody 15 0 35644 29M 2036 R 0.5 5.9 0:00 httpd
12817 nobody 13 0 19968 13M 2044 R 0.4 2.7 0:00 httpd
12833 nobody 14 0 19916 14M 2028 R 0.4 2.9 0:00 httpd
12851 nobody 14 0 19040 15M 2272 R 0.4 3.0 0:00 httpd
5 root 10 0 0 0 0 DW 0.3 0.0 0:01 kswapd
12164 nobody 12 0 24016 17M 1192 D 0.3 3.4 0:03 httpd
12186 nobody 14 0 26736 19M 1216 R 0.3 3.8 0:01 httpd
12573 nobody 14 0 10564 6556 1608 R 0.3 1.2 0:00 httpd
12706 nobody 14 0 8296 5312 1732 R 0.3 1.0 0:00 httpd
12829 nobody 10 0 19912 13M 1988 D 0.3 2.6 0:00 httpd
12873 nobody 14 0 12360 9024 2060 R 0.3 1.7 0:00 httpd
12372 root 11 0 640 572 324 R 0.2 0.1 0:01 top
12822 nobody 13 0 9884 6460 2288 R 0.2 1.2 0:00 httpd
11921 root 9 0 700 640 384 D 0.1 0.1 0:01 top
12178 nobody 9 0 7108 3548 1564 D 0.1 0.6 0:02 httpd
12180 nobody 10 0 19668 12M 1696 D 0.1 2.4 0:01 httpd
12628 nobody 11 0 17988 10M 1352 R 0.1 2.0 0:00 httpd
12654 nobody 10 0 14628 8160 1668 D 0.1 1.5 0:00 httpd
12696 nobody 9 0 59264 1832 660 D 0.1 0.3 0:01 httpd
12801 nobody 10 0 16064 12M 2112 D 0.1 2.4 0:00 httpd
12832 nobody 11 0 15392 11M 2096 D 0.1 2.1 0:00 httpd
12876 nobody 10 0 8180 4848 2228 D 0.1 0.9 0:00 httpd
12898 nobody 12 0 8248 5028 2184 D 0.1 0.9 0:00 httpd
12903 root 9 0 2404 2400 1944 D 0.1 0.4 0:00 exim
1 root 0 0 116 68 48 S 0.0 0.0 0:04 init
2 root 9 0 0 0 0 SW 0.0 0.0 0:00 keventd
3 root 9 0 0 0 0 SW 0.0 0.0 0:00 kapmd
4 root 19 19 0 0 0 SWN 0.0 0.0 0:01 ksoftirqd_CPU0
7 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush
8 root 9 0 0 0 0 SW 0.0 0.0 0:00 kupdated
9 root -1 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd
13 root 9 0 0 0 0 DW 0.0 0.0 0:01 kjournald
92 root 9 0 0 0 0 SW 0.0 0.0 0:00 khubd
219 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
653 root 9 0 244 212 156 D 0.0 0.0 0:00 syslogd
658 root 9 0 56 4 0 S 0.0 0.0 0:00 klogd
804 nobody 9 0 356 136 88 S 0.0 0.0 0:00 proftpd
822 root 8 0 216 4 0 S 0.0 0.0 0:00 sshd
[1]+ Stopped top 0 S 0.0 0.0 0:00 xinetd
kii75 root 9 0 416 352 224 S 0.0 0.0 0:00 antirelayd
-----------END RESULTS OF TOP-----------------

Any suggestions for getting this under control would be helpful. Thanks in advance, take care.

cPanel.net Support Ticket Number:

cPanel.net Support Ticket Number:
 

scottgem

Member
Feb 24, 2003
10
0
151
Well, this is getting me closer... check this out from CPU/Memory/MySQL Usage in WHM:

Top Process %CPU 99.9 httpd [www.XXXX.com] [/banners/aab323,7a823d636ade43315dd01e0ed08af1d2,3b121b.gif]
Top Process %CPU 54.0 httpd [www.XXXX.com] [/bannering.php?opver&cid122]
Top Process %CPU 49.0 httpd [www.XXXX.com] [/bannering.php?opver&cid110]


How the heck could a gif file, a banner for that matter, consume 99.9% of the CPU!?

cPanel.net Support Ticket Number:
 

scottgem

Member
Feb 24, 2003
10
0
151
Ah yes, well I just resolved the issue... but in the case that anyone is reading this horrible conversation i'm having with myself (sorry)... how could a gif file consume all of that CPU?

I've suspended the user and the load is 0.12 but now I have to figure out if this user was doing something malicious on purpose. Anyone ever hear of something like this? I'm calling it a gif-bomb if not, or maybe I can name it after myself.

cPanel.net Support Ticket Number:
 

Curious Too

Well-Known Member
Aug 31, 2001
432
1
318
cPanel Access Level
Root Administrator
Originally posted by scottgem
Ah yes, well I just resolved the issue... but in the case that anyone is reading this horrible conversation i'm having with myself (sorry)... how could a gif file consume all of that CPU?

I've suspended the user and the load is 0.12 but now I have to figure out if this user was doing something malicious on purpose. Anyone ever hear of something like this? I'm calling it a gif-bomb if not, or maybe I can name it after myself.

cPanel.net Support Ticket Number:
Could the user have been running a very popular banner exchange?

cPanel.net Support Ticket Number:
 

scottgem

Member
Feb 24, 2003
10
0
151
Yes, I just verified it.. it's definately a GIF and it's only 11k! I opened it up, viewed it, it looks clean as far as I can tell.

However, could there have been some type of endless loop that continually opened the gif? Even if that occurred how could that take the cpu away like that? I'm stumped. My client has since informed me that he was allowing people to upload banners, so now I feel that this uploaded GIF was in some way an intentional malicious act. I'm not sure if I can re-instate his account because frankly, I don't know what's going on!

It just doesn't add up... any ideas?

Thanks for your time.

cPanel.net Support Ticket Number:
 

Finkinstein

Well-Known Member
Mar 21, 2003
131
0
166
Someone could have uploaded it for maliscious purposes.
But, to create that much of a load by accesing it, hmm... have you checked the access logs? See if its been requested a lot?

cPanel.net Support Ticket Number:
 

scottgem

Member
Feb 24, 2003
10
0
151
Yeah, I checked that... it's been accessed under 25 times in the past 24 hours. This gets weirder and weirder.

cPanel.net Support Ticket Number:
 

pagedeveloping

Well-Known Member
Jun 11, 2003
219
0
166
New York
Perhaps!

I would look at this gif image real close to make sure there is not a link inbeded inside it. or even a javascript for that matter recalling the inbeded link.

It's been done before..

Pete

cPanel.net Support Ticket Number:
 

katz_global

Well-Known Member
PartnerNOC
it could also be an encrypted binary message, but still strange nevertheless.

It is possible to embed trojans in gifs that run stealth servers. It would actually be named .exe but be renamed .gif for the purpose so technically if they could ssh into the server and run this their hacks could be pulling in remotely across any number of other win systems.

Its a long shot, but possible.

The way to tell is download it and pass the file through a virus checker.