The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

HELP HELP!!!! Server Wide cPanel/WHM/Apache/MySQL Problems :(

Discussion in 'Database Discussions' started by mOdY, Feb 15, 2006.

  1. mOdY

    mOdY Well-Known Member

    Joined:
    Dec 25, 2004
    Messages:
    80
    Likes Received:
    0
    Trophy Points:
    6
    Hello all,

    I have massive flood of problems from one of my VPS servers running on Cent OS 4.2 x86_64 & having cPanel/WHM installed on it and Mysql 4.1.14 and Apache 1.3.34....

    First of all my main problems are with the loads on this server , it's too high even though it doesn't have more than 20 accounts on it, 2 of them are heavy loaded forums which have high MySQL usage but i'll list you my problems more technically down here.

    Apache seems like it's not closing the connection after it sent the user the requested data, here's my httpd.conf and /etc/my.cnf & below it some simple lines of my 'ps aux' as i can't paste the whole list here cause it's too large.

    [---------- Start httpd.conf -----------]
    RLimitMEM 355722581
    RLimitCPU 240
    ServerType standalone
    ServerRoot /usr/local/apache
    Timeout 22
    KeepAlive Off
    MaxKeepAliveRequests 1
    KeepAliveTimeout 1
    MinSpareServers 6
    MaxSpareServers 24
    StartServers 6
    MaxClients 768
    MaxRequestsPerChild 200
    [---------- End httpd.conf -----------]

    [---------- Start my.cnf -----------]
    [mysqld]
    set-variable = max_connections=500
    safe-show-database
    connect_timeout=8
    interactive_timeout=100
    max_connect_errors=8
    tmp_table_size=64M
    wait_timeout=15
    skip-innodb
    [mysqldump]
    quick
    max_allowed_packet=10M
    query_cache_size=128M
    query_cache_limit=2M
    query_cache_type=1
    key_buffer_size = 1G
    [---------- End my.cnf -----------]


    [---------- Start Some lines of 'ps aux' ----------]
    nobody 25197 1.0 0.5 123452 10376 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25198 6.7 1.1 131424 20868 ? S 14:49 0:01 /usr/local/apache/bin/httpd -DSSL
    nobody 25199 0.3 0.5 123316 10340 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25201 1.5 0.5 123316 10352 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25202 0.6 0.5 123316 10316 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25203 0.6 0.5 123312 10300 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25204 0.3 0.5 123316 10332 ? R 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25208 0.4 0.5 123316 10344 ? R 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25209 2.3 0.0 0 0 ? Z 14:49 0:00 [httpd] <defunct>
    mysql 25217 15.1 0.8 48176 15908 pts/2 S 14:49 0:01 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
    mysql 25219 10.1 0.8 47980 15888 pts/2 R 14:49 0:01 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
    nobody 25220 10.5 0.9 126556 16708 ? S 14:49 0:01 /usr/local/apache/bin/httpd -DSSL
    nobody 25222 1.1 0.5 127320 10528 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25223 8.2 0.9 127468 16760 ? R 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    mysql 25224 3.2 0.8 47980 15888 pts/2 S 14:49 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
    mysql 25226 6.1 0.8 47980 15888 pts/2 R 14:49 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
    nobody 25231 4.9 0.5 123452 10388 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    mysql 25232 4.0 0.8 47980 15888 pts/2 R 14:49 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
    nobody 25233 7.1 0.9 127716 17484 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25234 15.0 1.1 131084 21468 ? S 14:49 0:01 /usr/local/apache/bin/httpd -DSSL
    nobody 25235 3.4 0.5 123452 10412 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25239 2.0 0.5 123452 10380 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25240 3.7 0.6 123720 11100 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25241 13.2 1.1 131184 21544 ? S 14:49 0:01 /usr/local/apache/bin/httpd -DSSL
    nobody 25242 0.7 0.5 123316 10296 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25243 10.1 0.9 126712 16844 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25245 11.3 0.9 127736 17420 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25246 6.5 0.0 0 0 ? Z 14:49 0:00 [httpd] <defunct>
    nobody 25247 15.7 0.8 125976 15548 ? S 14:49 0:01 /usr/local/apache/bin/httpd -DSSL
    nobody 25254 4.7 0.6 123744 11140 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25256 2.8 0.5 123316 10368 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25257 8.2 0.9 127728 17456 ? R 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25260 2.0 0.5 123452 10368 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25261 19.2 0.0 0 0 ? Z 14:49 0:00 [httpd] <defunct>
    nobody 25262 8.2 0.5 123452 10380 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25263 1.6 0.5 123316 10340 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25264 11.8 0.9 127716 17440 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25265 6.0 0.5 123316 10348 ? R 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25266 1.8 0.5 123316 10336 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25273 4.2 0.5 123316 10336 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25278 7.4 0.6 123808 11208 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25279 6.5 0.5 123316 10340 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25281 15.5 0.5 123448 10380 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25282 38.5 1.1 131264 21072 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25283 19.0 0.5 123316 10380 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    mysql 25284 23.0 0.0 0 0 pts/2 Z 14:49 0:00 [mysqld] <defunct>
    nobody 25289 3.5 0.5 123316 10336 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25290 24.0 0.0 0 0 ? Z 14:49 0:00 [httpd] <defunct>
    nobody 25291 10.0 0.5 123312 10368 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    mysql 25292 23.0 0.8 47980 15888 pts/2 S 14:49 0:00 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=
    nobody 25293 24.0 0.5 123452 10388 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25294 23.0 0.6 123716 10996 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25295 9.0 0.5 123312 10344 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25296 34.0 0.5 123452 10388 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25297 49.0 0.5 123316 10384 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25298 8.0 0.5 123316 10332 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25299 43.0 0.0 0 0 ? Z 14:49 0:00 [httpd] <defunct>
    nobody 25300 9.0 0.6 123708 11004 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25301 0.0 0.5 123316 10332 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25302 0.0 0.5 123316 10328 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25303 0.0 0.5 123316 10332 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    nobody 25304 0.0 0.5 123312 10360 ? S 14:49 0:00 /usr/local/apache/bin/httpd -DSSL
    [---------- End Some lines of 'ps aux' ----------]


    [---------- Start Server's Uptime ----------]
    14:53:03 up 5 days, 20:42, 2 users, load average: 26.56, 24.22, 24.39
    [---------- End Server's Uptime ----------]


    [---------- Start Top ----------]
    top - 15:14:00 up 5 days, 21:03, 2 users, load average: 32.34, 25.73, 24.34
    Tasks: 419 total, 22 running, 372 sleeping, 1 stopped, 24 zombie
    Cpu(s): 34.7% us, 16.3% sy, 0.0% ni, 40.9% id, 6.8% wa, 0.3% hi, 1.0% si
    Mem: 1812452k total, 1751924k used, 60528k free, 23812k buffers
    Swap: 3534260k total, 207504k used, 3326756k free, 849544k cached
    [---------- End Top ----------]


    [---------- Start some netstat info ----------]
    root@server [~]# netstat -nt | grep :80 | wc -l
    1390
    [---------- End some netstat info ----------]


    Note: I have Zend Optimizer 2.6.2 for x86_64 & have eaccelerator-0.9.4-rc1 & have FastCGI Apache module.

    Also i've suspended the most using account as to see if there's a change and not a clue....

    Please if someone think he could share finding this problem it'll be most welcomed :)
     
  2. webignition

    webignition Well-Known Member

    Joined:
    Jan 22, 2005
    Messages:
    1,880
    Likes Received:
    0
    Trophy Points:
    36
    The ps output shows a fair amount of Apache processes - is this normal for this server, or is it more than you would expect?

    I recently had a problem with a user who was using Mambo with a bit of a dodgy modified template. The symptoms were loads of Apache processes and looking at the Apache Status page in WHM revealed that the IP of the requesting machine was the server itself.

    Each request to the Mambo-based site was spawning a request by Mambo itself for something within Mambo, and this request spawned another identical request and so on.

    It's quite easy to see how one request to the Mambo-based site was nearly killing the server in less than a second.
     
  3. mOdY

    mOdY Well-Known Member

    Joined:
    Dec 25, 2004
    Messages:
    80
    Likes Received:
    0
    Trophy Points:
    6
    Well, this ps output isn't even half of the real ps output im having although pasting it here will be loose of time & i don't have any mambo hosted on this vps although it's a good thing to paste here an httpd status maybe it could help in tracing the core of the problem...

    Server Built: Feb 13 2006 20:26:46
    _________________________________________________________________

    Current Time: Wednesday, 15-Feb-2006 15:49:34 EST
    Restart Time: Wednesday, 15-Feb-2006 15:40:15 EST
    Parent Server Generation: 5
    Server uptime: 9 minutes 19 seconds
    Total accesses: 8595 - Total Traffic: 224.9 MB
    CPU Usage: u116.47 s24.52 cu.09 cs.11 - 25.3% CPU load
    15.4 requests/sec - 411.9 kB/second - 26.8 kB/request
    219 requests currently being processed, 7 idle servers
    WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWWWWWWWWWWWWWWWW
    _WWWWWWWWW_WWWWWW_WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_WWWWWWWW
    WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW_LWWWWWWW
    WWWWWWWWWWWWWW_WWWSWWWW....W..W...W.WWW.W.WW.....WW.............
    ................................................................
    ................................................................
    ................................................................
    ................................................................
    ................................................................
    ................................................................
    ................................................................
     
  4. webignition

    webignition Well-Known Member

    Joined:
    Jan 22, 2005
    Messages:
    1,880
    Likes Received:
    0
    Trophy Points:
    36
    Well, I can see that there are a fair few active requests.

    Is there anything common about them? Look for patterns to find the cause!
     
  5. compunet2

    compunet2 Well-Known Member

    Joined:
    Feb 21, 2003
    Messages:
    310
    Likes Received:
    0
    Trophy Points:
    16
    What Kernel version are you using?
     
  6. mOdY

    mOdY Well-Known Member

    Joined:
    Dec 25, 2004
    Messages:
    80
    Likes Received:
    0
    Trophy Points:
    6

    Look for patterns ! how ?
     
  7. mOdY

    mOdY Well-Known Member

    Joined:
    Dec 25, 2004
    Messages:
    80
    Likes Received:
    0
    Trophy Points:
    6

    Kernel 2.6.12.6 x86_64
     
  8. acenetryan

    acenetryan Well-Known Member
    PartnerNOC

    Joined:
    Aug 21, 2005
    Messages:
    197
    Likes Received:
    1
    Trophy Points:
    18
    In apache status, you can look for a large amount of requests from a single IP. If you see an abnormal amount, use iptables to block the IP. You may also want to look into "netstat". It can show you current network traffic and is helpful if you're trying to find the cause of a DOS.
     
Loading...

Share This Page