The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HUGE Load, Runaway server, what is happening!

Discussion in 'General Discussion' started by scottgem, Jun 26, 2003.

  1. scottgem

    scottgem Member

    Joined:
    Feb 24, 2003
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    I'm having a problem with one of my servers! It keeps running itself into the ground... I'm completely stumped. It's up for a matter of 40 minutes and the load spikes to 129.00+ ! If the httpd processes are not killed its gone and it must be manually rebooted at the NOC.

    I just regained control of it by luckily getting a kill httpd command in... my result after it coming back to life is a load average: 241.50, 261.48, 183.51

    What are the possibilities for this? The server is definately loaded but I don't think thats the true issue, take a look at the mem report below. It doesn't make much sense to me because it displays the load at 129.00 but cpu is 91.7% idle!? How is this possible? The server is fully updated, with RHN as well as C-Panel stable release. Newest apache core installed as well. Kernel is up to date (2.4.20-18.7)

    -----------RESULTS OF TOP-----------------
    12:07pm up 52 min, 2 users, load average: 129.00, 70.78, 39.22
    257 processes: 230 sleeping, 25 running, 2 zombie, 0 stopped
    CPU states: 2.9% user, 5.2% system, 0.0% nice, 91.7% idle
    Mem: 514120K av, 507772K used, 6348K free, 0K shrd, 5068K buff
    Swap: 2048276K av, 1111316K used, 936960K free 27292K cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
    12454 nobody 15 0 20596 15M 1672 R 0.7 3.1 0:00 httpd
    12461 nobody 18 0 9904 5056 1068 R 0.7 0.9 0:00 httpd
    6 root 12 0 0 0 0 SW 0.5 0.0 0:17 kscand
    12821 nobody 15 0 35644 29M 2036 R 0.5 5.9 0:00 httpd
    12817 nobody 13 0 19968 13M 2044 R 0.4 2.7 0:00 httpd
    12833 nobody 14 0 19916 14M 2028 R 0.4 2.9 0:00 httpd
    12851 nobody 14 0 19040 15M 2272 R 0.4 3.0 0:00 httpd
    5 root 10 0 0 0 0 DW 0.3 0.0 0:01 kswapd
    12164 nobody 12 0 24016 17M 1192 D 0.3 3.4 0:03 httpd
    12186 nobody 14 0 26736 19M 1216 R 0.3 3.8 0:01 httpd
    12573 nobody 14 0 10564 6556 1608 R 0.3 1.2 0:00 httpd
    12706 nobody 14 0 8296 5312 1732 R 0.3 1.0 0:00 httpd
    12829 nobody 10 0 19912 13M 1988 D 0.3 2.6 0:00 httpd
    12873 nobody 14 0 12360 9024 2060 R 0.3 1.7 0:00 httpd
    12372 root 11 0 640 572 324 R 0.2 0.1 0:01 top
    12822 nobody 13 0 9884 6460 2288 R 0.2 1.2 0:00 httpd
    11921 root 9 0 700 640 384 D 0.1 0.1 0:01 top
    12178 nobody 9 0 7108 3548 1564 D 0.1 0.6 0:02 httpd
    12180 nobody 10 0 19668 12M 1696 D 0.1 2.4 0:01 httpd
    12628 nobody 11 0 17988 10M 1352 R 0.1 2.0 0:00 httpd
    12654 nobody 10 0 14628 8160 1668 D 0.1 1.5 0:00 httpd
    12696 nobody 9 0 59264 1832 660 D 0.1 0.3 0:01 httpd
    12801 nobody 10 0 16064 12M 2112 D 0.1 2.4 0:00 httpd
    12832 nobody 11 0 15392 11M 2096 D 0.1 2.1 0:00 httpd
    12876 nobody 10 0 8180 4848 2228 D 0.1 0.9 0:00 httpd
    12898 nobody 12 0 8248 5028 2184 D 0.1 0.9 0:00 httpd
    12903 root 9 0 2404 2400 1944 D 0.1 0.4 0:00 exim
    1 root 0 0 116 68 48 S 0.0 0.0 0:04 init
    2 root 9 0 0 0 0 SW 0.0 0.0 0:00 keventd
    3 root 9 0 0 0 0 SW 0.0 0.0 0:00 kapmd
    4 root 19 19 0 0 0 SWN 0.0 0.0 0:01 ksoftirqd_CPU0
    7 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush
    8 root 9 0 0 0 0 SW 0.0 0.0 0:00 kupdated
    9 root -1 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd
    13 root 9 0 0 0 0 DW 0.0 0.0 0:01 kjournald
    92 root 9 0 0 0 0 SW 0.0 0.0 0:00 khubd
    219 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald
    653 root 9 0 244 212 156 D 0.0 0.0 0:00 syslogd
    658 root 9 0 56 4 0 S 0.0 0.0 0:00 klogd
    804 nobody 9 0 356 136 88 S 0.0 0.0 0:00 proftpd
    822 root 8 0 216 4 0 S 0.0 0.0 0:00 sshd
    [1]+ Stopped top 0 S 0.0 0.0 0:00 xinetd
    kii75 root 9 0 416 352 224 S 0.0 0.0 0:00 antirelayd
    -----------END RESULTS OF TOP-----------------

    Any suggestions for getting this under control would be helpful. Thanks in advance, take care.

    cPanel.net Support Ticket Number:

    cPanel.net Support Ticket Number:
     
  2. scottgem

    scottgem Member

    Joined:
    Feb 24, 2003
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    Well, this is getting me closer... check this out from CPU/Memory/MySQL Usage in WHM:

    Top Process %CPU 99.9 httpd [www.XXXX.com] [/banners/aab323,7a823d636ade43315dd01e0ed08af1d2,3b121b.gif]
    Top Process %CPU 54.0 httpd [www.XXXX.com] [/bannering.php?opver&cid122]
    Top Process %CPU 49.0 httpd [www.XXXX.com] [/bannering.php?opver&cid110]


    How the heck could a gif file, a banner for that matter, consume 99.9% of the CPU!?

    cPanel.net Support Ticket Number:
     
  3. scottgem

    scottgem Member

    Joined:
    Feb 24, 2003
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    Ah yes, well I just resolved the issue... but in the case that anyone is reading this horrible conversation i'm having with myself (sorry)... how could a gif file consume all of that CPU?

    I've suspended the user and the load is 0.12 but now I have to figure out if this user was doing something malicious on purpose. Anyone ever hear of something like this? I'm calling it a gif-bomb if not, or maybe I can name it after myself.

    cPanel.net Support Ticket Number:
     
  4. Curious Too

    Curious Too Well-Known Member

    Joined:
    Aug 31, 2001
    Messages:
    427
    Likes Received:
    0
    Trophy Points:
    16
    cPanel Access Level:
    Root Administrator
    Could the user have been running a very popular banner exchange?

    cPanel.net Support Ticket Number:
     
  5. Finkinstein

    Finkinstein Well-Known Member

    Joined:
    Mar 21, 2003
    Messages:
    131
    Likes Received:
    0
    Trophy Points:
    16
    That would have to be VERY popular.

    cPanel.net Support Ticket Number:
     
  6. Juanra

    Juanra Well-Known Member

    Joined:
    Sep 22, 2001
    Messages:
    777
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Spain
    Are you sure it's a gif file and not a php script? Is there an actual gif file with that name in that directory?

    cPanel.net Support Ticket Number: no thanks
     
  7. scottgem

    scottgem Member

    Joined:
    Feb 24, 2003
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    Yes, I just verified it.. it's definately a GIF and it's only 11k! I opened it up, viewed it, it looks clean as far as I can tell.

    However, could there have been some type of endless loop that continually opened the gif? Even if that occurred how could that take the cpu away like that? I'm stumped. My client has since informed me that he was allowing people to upload banners, so now I feel that this uploaded GIF was in some way an intentional malicious act. I'm not sure if I can re-instate his account because frankly, I don't know what's going on!

    It just doesn't add up... any ideas?

    Thanks for your time.

    cPanel.net Support Ticket Number:
     
  8. Finkinstein

    Finkinstein Well-Known Member

    Joined:
    Mar 21, 2003
    Messages:
    131
    Likes Received:
    0
    Trophy Points:
    16
    Someone could have uploaded it for maliscious purposes.
    But, to create that much of a load by accesing it, hmm... have you checked the access logs? See if its been requested a lot?

    cPanel.net Support Ticket Number:
     
  9. scottgem

    scottgem Member

    Joined:
    Feb 24, 2003
    Messages:
    10
    Likes Received:
    0
    Trophy Points:
    1
    Yeah, I checked that... it's been accessed under 25 times in the past 24 hours. This gets weirder and weirder.

    cPanel.net Support Ticket Number:
     
  10. pagedeveloping

    pagedeveloping Well-Known Member

    Joined:
    Jun 11, 2003
    Messages:
    219
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    New York
    Perhaps!

    I would look at this gif image real close to make sure there is not a link inbeded inside it. or even a javascript for that matter recalling the inbeded link.

    It's been done before..

    Pete

    cPanel.net Support Ticket Number:
     
  11. katz_global

    katz_global Well-Known Member
    PartnerNOC

    Joined:
    Oct 18, 2003
    Messages:
    69
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Hosting from: Panama, Hong Kong, Singapore, Malays
    it could also be an encrypted binary message, but still strange nevertheless.

    It is possible to embed trojans in gifs that run stealth servers. It would actually be named .exe but be renamed .gif for the purpose so technically if they could ssh into the server and run this their hacks could be pulling in remotely across any number of other win systems.

    Its a long shot, but possible.

    The way to tell is download it and pass the file through a virus checker.
     
Loading...

Share This Page