The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

High Server Load Problem

Discussion in 'General Discussion' started by iwebserv, May 27, 2004.

  1. iwebserv

    iwebserv Member

    Joined:
    May 14, 2004
    Messages:
    18
    Likes Received:
    0
    Trophy Points:
    1
    I am having issues with one particular customer of mine, every night same thing all the time, server load tends to jump sky high and then settles again. %CPU is 1.07 and %MEM gets to 37.27 and these three items run under their domain in the CPU/Memory/MySQL Usage

    Top Process %CPU 36.0 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --pid-file/var/lib/mysql/net1-sgstx.online-network.net.pid --skip-locking --skip-locking --log --log-update
    Top Process %CPU 35.0 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --pid-file/var/lib/mysql/net1-sgstx.online-network.net.pid --skip-locking --skip-locking --log --log-update
    Top Process %CPU 19.0 /usr/sbin/mysqld --basedir/ --datadir/var/lib/mysql --usermysql --pid-file/var/lib/mysql/net1-sgstx.online-network.net.pid --skip-locking --skip-locking --log --log-update

    Under the CPU/Memory/MySQL Usage, there is another something that is running that takes quite a bit of load as well:

    Top Process %CPU 14.0 /usr/bin/perl /usr/bin/mrtg /etc/mrtg/mrtg.cfg
    Top Process %CPU 13.0 /usr/bin/perl /usr/bin/mrtg /etc/mrtg/mrtg.cfg
    Top Process %CPU 4.0 /usr/bin/perl /usr/bin/mrtg /etc/mrtg/mrtg.cfg

    Does anyone know what these are and how they work and what is behind them going so high? The customer runs a large phpbb1 sql database with just under 7,000 registered members. And each night during peak times the server tends to get these high server loads.
     
  2. iwebserv

    iwebserv Member

    Joined:
    May 14, 2004
    Messages:
    18
    Likes Received:
    0
    Trophy Points:
    1
    anyone know?
     
  3. jayntguru

    jayntguru Member

    Joined:
    May 14, 2004
    Messages:
    17
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Acworth, GA USA
    I found the same thing this morning. Not sure what is up.
     
  4. JBapt

    JBapt Active Member

    Joined:
    Feb 5, 2004
    Messages:
    41
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Porto - Portugal
    I believe that can be normal. What is aparently causing you high loads is MySQL wich as you know "can" (might not be) using MySQL. Also, phpbb scripts may not be very optimized. Just add all your other sites to the load this one generates and this can be the cause. This can even be worst if your daily backup kicks in during this high loads. But to be safe, and if this happens every single night... just ask your costumer if he can put the forum offline for a couple of hours to check things out. I believe this can be an fair request because the good maintenance of the server is the good maintenance of his forum.
     
  5. Bruce

    Bruce Well-Known Member

    Joined:
    Oct 4, 2001
    Messages:
    147
    Likes Received:
    0
    Trophy Points:
    16
    I get the same thing at anytime of day.
    When this happens, I re-start MySQL server and goes away for a few days,
    Anybody have a real solution to fix this problem ?
     
  6. JBapt

    JBapt Active Member

    Joined:
    Feb 5, 2004
    Messages:
    41
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Porto - Portugal
    Update your MySQL version.
    If the problem continues try to isolate the problem. Get to know what users are doing queries.
     
  7. Bruce

    Bruce Well-Known Member

    Joined:
    Oct 4, 2001
    Messages:
    147
    Likes Received:
    0
    Trophy Points:
    16
    They are lining up again.

    /usr/sbin/mysqld--basedir=/--datadir=/var/lib/mysql--user=mysql--pid-file=/var/lib/mysql/taurus.memtek.us.pid--skip-locking

    Did a trace on one of them, have no idea what this is doing.


    select(5, [3 4], NULL, NULL, NULL) = 1 (in [4])
    fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    accept(4, {sin_family=AF_UNIX, path="ÿ¿†Z#u"}, [2]) = 1736
    fcntl64(4, F_SETFL, O_RDWR) = 0
    getsockname(1736, {sin_family=AF_UNIX, path="/var/lib/mysql"}, [28]) = 0
    fcntl64(1736, F_GETFL) = 0x2 (flags O_RDWR)
    fcntl64(1736, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    setsockopt(1736, SOL_IP, IP_TOS, [8], 4) = -1 EOPNOTSUPP (Operation not supported)
    kill(28972, SIGRTMIN) = 0
    kill(28972, SIGRTMIN) = 0
    select(5, [3 4], NULL, NULL, NULL) = 1 (in [4])
    fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    accept(4, {sin_family=AF_UNIX, path="ÿ¿†Z#u"}, [2]) = 1736
    fcntl64(4, F_SETFL, O_RDWR) = 0
    getsockname(1736, {sin_family=AF_UNIX, path="/var/lib/mysql"}, [28]) = 0
    fcntl64(1736, F_GETFL) = 0x2 (flags O_RDWR)
    fcntl64(1736, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    setsockopt(1736, SOL_IP, IP_TOS, [8], 4) = -1 EOPNOTSUPP (Operation not supported)
    kill(1878, SIGRTMIN) = 0
    kill(1878, SIGRTMIN) = 0
    select(5, [3 4], NULL, NULL, NULL) = 1 (in [4])
    fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    accept(4, {sin_family=AF_UNIX, path="ÿ¿†Z#u"}, [2]) = 1736
    fcntl64(4, F_SETFL, O_RDWR) = 0
    getsockname(1736, {sin_family=AF_UNIX, path="/var/lib/mysql"}, [28]) = 0
    fcntl64(1736, F_GETFL) = 0x2 (flags O_RDWR)
    fcntl64(1736, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    setsockopt(1736, SOL_IP, IP_TOS, [8], 4) = -1 EOPNOTSUPP (Operation not supported)
    kill(1876, SIGRTMIN) = 0
    kill(1876, SIGRTMIN) = 0
    select(5, [3 4], NULL, NULL, NULL) = 1 (in [4])
    fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    accept(4, {sin_family=AF_UNIX, path="ÿ¿†Z#u"}, [2]) = 1736
    fcntl64(4, F_SETFL, O_RDWR) = 0
    getsockname(1736, {sin_family=AF_UNIX, path="/var/lib/mysql"}, [28]) = 0
    fcntl64(1736, F_GETFL) = 0x2 (flags O_RDWR)
    fcntl64(1736, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    setsockopt(1736, SOL_IP, IP_TOS, [8], 4) = -1 EOPNOTSUPP (Operation not supported)
    kill(28507, SIGRTMIN) = 0
    kill(28507, SIGRTMIN) = 0
    select(5, [3 4], NULL, NULL, NULL) = 1 (in [4])
    fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    accept(4, {sin_family=AF_UNIX, path="ÿ¿†Z#u"}, [2]) = 1736
    fcntl64(4, F_SETFL, O_RDWR) = 0
    getsockname(1736, {sin_family=AF_UNIX, path="/var/lib/mysql"}, [28]) = 0
    fcntl64(1736, F_GETFL) = 0x2 (flags O_RDWR)
    fcntl64(1736, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    setsockopt(1736, SOL_IP, IP_TOS, [8], 4) = -1 EOPNOTSUPP (Operation not supported)
    kill(28929, SIGRTMIN) = 0
    kill(28929, SIGRTMIN) = 0
    select(5, [3 4], NULL, NULL, NULL) = 1 (in [4])
    fcntl64(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    accept(4, {sin_family=AF_UNIX, path="ÿ¿†Z#u"}, [2]) = 1736
    fcntl64(4, F_SETFL, O_RDWR) = 0
    getsockname(1736, {sin_family=AF_UNIX, path="/var/lib/mysql"}, [28]) = 0
    fcntl64(1736, F_GETFL) = 0x2 (flags O_RDWR)
    fcntl64(1736, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    setsockopt(1736, SOL_IP, IP_TOS, [8], 4) = -1 EOPNOTSUPP (Operation not supported)
     
  8. Bruce

    Bruce Well-Known Member

    Joined:
    Oct 4, 2001
    Messages:
    147
    Likes Received:
    0
    Trophy Points:
    16
    Here is another trace

    --- SIGRTMIN (Real-time signal 0) ---
    sigreturn() = ? (mask now [HUP INT QUIT PIPE TERM TSTP RTMIN])
    setsockopt(7, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
    write(7, "0\0\0\0\n4.0.20-standard\0\16\0\0\0[2(p|>`"..., 52) = 52
    read(7, 0x84696e8, 4) = -1 EAGAIN (Resource temporarily unavailable)
    time(NULL) = 1091553327
    rt_sigprocmask(SIG_BLOCK, ~[33], [HUP INT QUIT PIPE TERM TSTP RTMIN], 8) = 0
    kill(5460, SIGALRM) = 0
    kill(5460, SIGRTMIN) = 0
    rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT PIPE TERM TSTP RTMIN], NULL, 8) = 0
    fcntl64(7, F_SETFL, O_RDWR) = 0
    read(7, "\34\0\0\1", 4) = 4
    read(7, "\5 \0\0\0rickart_pchat1\0HYSP_KFW", 28) = 28
    rt_sigprocmask(SIG_BLOCK, ~[33], [HUP INT QUIT PIPE TERM TSTP RTMIN], 8) = 0
    rt_sigprocmask(SIG_SETMASK, [HUP INT QUIT PIPE TERM TSTP RTMIN], NULL, 8) = 0
    fcntl64(7, F_SETFL, O_RDWR|O_NONBLOCK) = 0
    write(7, "\3\0\0\2\0\0\0", 7) = 7
    time([1091553327]) = 1091553327
    read(7, 0x84696e8, 4) = -1 EAGAIN (Resource temporarily unavailable)
     
  9. jester.ro

    jester.ro Well-Known Member
    PartnerNOC

    Joined:
    Feb 6, 2004
    Messages:
    304
    Likes Received:
    0
    Trophy Points:
    16
    Location:
    Bucharest, Romania
    cPanel Access Level:
    DataCenter Provider
    check the mysql process list.
    for this, go in you phpmyadmin from whm, and click 'show processess"


    if you se alot of processes in sleep state, it's probably phpbb's fault.

    older version(dunno about new ones) used mysql_pconnect, that made a persistent connection to the mysql server, and for some reson, did not clise them.


    if that is the case, find you php.ini file, search for mysql.allow_persistent, and swtich it to off, then restart apache.

    this way, mysql_pconnect falls back to mysql-connect, which is more memory-friendly.

    then you could try optimizing you mysql.
    search on the server for mysql-large.cnf. go to that dir, and there you'll find 4 working examples of my.cnf files

    read each one (mysql-medium, mysql-large, mysql-huge) especially the beginning, where it says for wich kind of setup is suited each file.

    if, for example, mysql-large.cnf is suited for you, bakup /etc/my.cnf somewhere, and copy mysql-large.cnf over /etc/my.cnf.

    restart mysql, and watch the load.
     
  10. Jeff-C

    Jeff-C Well-Known Member

    Joined:
    Mar 16, 2004
    Messages:
    117
    Likes Received:
    0
    Trophy Points:
    16
    Same problem here. For now I've changed to incremental backups (even though I would really prefer to do automated compressed backups.)

    Twice now compressed backups have caused a real problem. Load went from 0.20 (15-minute) to over 40 and remained that way for over 30 minutes, with iowait being 99% while the backup script is creating the compressed archives. This then causes mysql processes to really stack up, and after about 15 minutes of the backup process starting, forums and other mysql powered sites are no longer loading. Total data being backed up is about 8 GB, but with two sites around 1.5 GB each. Server is a 2.8 Ghz HT server with 1 GB of Ram, latest cpanel stable, and RHE OS.
     
  11. bigpy2003

    bigpy2003 Well-Known Member

    Joined:
    Feb 22, 2004
    Messages:
    124
    Likes Received:
    0
    Trophy Points:
    16
    Well, I hosted a fairly large site as well. 100 constant people on at all times, with 65k registered members. The server only really sat at 4...and trust me, the site was programmed very badly.

    You don't have to completely disable pconnect's, however it does help you set a lower timeout for them. I believe at default they expire in 60 mins (not sure)...I set mine to expire in 300 secs (5 mins). My machine is only a 2.4ghz celeron with 512mb ram, so I don't see why you guys are having this problem.

    I also suggest using phpmyadmin to see what queries are being ran.

    Or you can do it from command line.

    mysql show processlist;

    i believe is the command.

    Maybe give us a screenshot of the 'top' command to see what else might be causing your load.
     
Loading...

Share This Page