The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Poss memory leak ?

Discussion in 'General Discussion' started by fuzioneer, Sep 9, 2005.

  1. fuzioneer

    fuzioneer Well-Known Member

    Joined:
    Dec 12, 2003
    Messages:
    98
    Likes Received:
    0
    Trophy Points:
    6
    We have centos 3.4 running on our server.

    We have started encountering random hangs

    Upon investgation it appears that server hangs are caused by slowly building memory usage until eventually server falls over.

    My linux guy has been running some scripts to track memory usage and it appears to be MySql that is slowly but surely eating up ram

    We have a lot of phpnuke and forums running on this particular server so mysql does get hammered.

    relevant snippet from my.cnf:-

    # The MySQL server
    [mysqld]
    port = 3306
    socket = /var/lib/mysql/mysql.sock
    skip-locking
    key_buffer = 384M
    max_allowed_packet = 6M
    table_cache = 512
    sort_buffer_size = 2M
    read_buffer_size = 2M
    myisam_sort_buffer_size = 64M
    thread_cache = 8
    query_cache_size = 32M
    # Try number of CPU's*2 for thread_concurrency
    thread_concurrency = 8


    He wrote a small perl script to ascertain mysql process memory usage as follows:-

    #!/usr/bin/perl

    $date = scalar localtime;
    $ps = `ps -eo pid,user,rss,args`;
    @procs = split /\n/, $ps;

    open LOGF, ">>/var/tmp/process_mem.log";
    print LOGF "\n$date\n";

    foreach $proc (@procs) {
    chomp $proc;
    @a = split /\s+/, $proc;
    if (@a[2] >= 71680) {
    print LOGF "@a[3] (@a[0]) using more than 70mb memory - (@a[2] kb)\n";
    }
    }

    print LOGF "\n*******


    initially the log had entries like this from last night after a reboot:-
    Thu Sep 8 23:30:00 2005
    /usr/sbin/mysqld (14878) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (15946) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (20824) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (10875) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (13544) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (13607) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (14470) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (15814) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (16617) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (17514) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (19699) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (19858) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (19862) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (21547) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (23329) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (23470) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (23473) using more than 70mb memory - (94584 kb)
    /usr/sbin/mysqld (23624) using more than 70mb memory - (94584 kb)


    at present time this is already up to the following:-
    Fri Sep 9 10:30:00 2005
    /usr/sbin/mysqld (20824) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (19858) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (26125) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (11562) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (17485) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (19113) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (19340) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (22260) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (23813) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (25843) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (26635) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (26636) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (26637) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (26638) using more than 70mb memory - (139728 kb)
    /usr/sbin/mysqld (26639) using more than 70mb memory - (139728 kb)

    and indications are this will just increase over time until next crash.

    free command result:-
    free
    total used free shared buffers cached
    Mem: 2045964 1581428 464536 0 82544 442620
    -/+ buffers/cache: 1056264 989700
    Swap: 2040244 198252 1841992

    Below is the MRTG swap file stats for the server
    [​IMG]
    HELP :)
    ANyone got any ideas as to why this is happening ?
     
  2. xixi

    xixi Active Member

    Joined:
    Mar 11, 2004
    Messages:
    28
    Likes Received:
    0
    Trophy Points:
    6
    Any progress in that case ?
    For a few days on my server i also noticed the same problem, mysqld is eating up all avalible memory and after 2-3 hours server is hanging out.
    How can i check which database is eatng it up ?
     
  3. positive

    positive Member

    Joined:
    Mar 29, 2003
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    i have the same problem.
    any solutions?
     
  4. abubin

    abubin Well-Known Member

    Joined:
    Dec 7, 2004
    Messages:
    393
    Likes Received:
    1
    Trophy Points:
    18
    me too have mysqld eating up memory. I have lot and lots of mysql processes running with each taking like 70mb memory. But for my case, I need to restart mysql after a week or less. But still it's really annoying problem.

    Please help...
     
  5. abubin

    abubin Well-Known Member

    Joined:
    Dec 7, 2004
    Messages:
    393
    Likes Received:
    1
    Trophy Points:
    18
    here is the problem with top screenshot :


    12:26:09 up 83 days, 2:09, 1 user, load average: 0.50, 0.29, 0.21
    226 processes: 223 sleeping, 1 running, 2 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 14.8% 0.0% 1.9% 0.0% 0.0% 4.4% 78.7%
    cpu00 5.5% 0.0% 1.8% 0.0% 0.0% 0.0% 92.5%
    cpu01 1.8% 0.0% 1.8% 0.0% 0.0% 0.0% 96.2%
    Mem: 1015908k av, 938472k used, 77436k free, 0k shrd, 57860k buff
    682480k actv, 139464k in_d, 13656k in_c
    Swap: 2048276k av, 176212k used, 1872064k free 361744k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    26798 mysql 15 0 74128 70M 1608 S 0.0 7.1 0:21 1 mysqld
    26799 mysql 15 0 74128 70M 1608 S 0.0 7.1 0:02 0 mysqld
    26800 mysql 15 0 74128 70M 1608 S 0.0 7.1 0:12 0 mysqld
    26801 mysql 15 0 74128 70M 1608 S 0.0 7.1 7:57 0 mysqld
    26809 mysql 15 0 74128 70M 1608 S 0.0 7.1 7:57 1 mysqld
    26923 mysql 15 0 74128 70M 1608 S 0.0 7.1 8:05 0 mysqld
    27098 mysql 15 0 74128 70M 1608 S 0.0 7.1 6:48 0 mysqld
    27306 mysql 15 0 74128 70M 1608 S 0.0 7.1 7:50 1 mysqld
    27321 mysql 15 0 74128 70M 1608 S 0.0 7.1 6:55 1 mysqld
    29520 mysql 15 0 74128 70M 1608 S 0.0 7.1 7:37 1 mysqld
    8313 mysql 15 0 74128 70M 1608 S 0.0 7.1 6:31 0 mysqld
    22313 mysql 15 0 74128 70M 1608 S 0.0 7.1 6:12 0 mysqld
    22314 mysql 15 0 74128 70M 1608 S 0.0 7.1 6:19 1 mysqld
    9380 mysql 15 0 74128 70M 1608 S 0.0 7.1 4:59 1 mysqld
    9381 mysql 15 0 74128 70M 1608 S 0.0 7.1 5:10 1 mysqld
    9382 mysql 15 0 74128 70M 1608 S 0.0 7.1 5:19 0 mysqld
    9383 mysql 15 0 74128 70M 1608 S 0.0 7.1 5:00 0 mysqld
    9384 mysql 15 0 74128 70M 1608 S 0.0 7.1 4:47 0 mysqld
    9386 mysql 15 0 74128 70M 1608 S 0.0 7.1 4:45 0 mysqld
    9387 mysql 15 0 74128 70M 1608 S 0.0 7.1 4:54 1 mysqld
    9388 mysql 16 0 74128 70M 1608 S 0.0 7.1 5:04 0 mysqld
    9396 mysql 15 0 74128 70M 1608 S 0.0 7.1 5:46 1 mysqld
    19105 mysql 15 0 74128 70M 1608 S 0.0 7.1 3:39 1 mysqld
    19106 mysql 15 0 74128 70M 1608 S 0.0 7.1 3:24 0 mysqld
    19107 mysql 15 0 74128 70M 1608 S 0.0 7.1 2:46 0 mysqld


    This is really crazy....anyone have any idea what is happening?
     
  6. WestBend

    WestBend Well-Known Member

    Joined:
    Oct 12, 2003
    Messages:
    173
    Likes Received:
    0
    Trophy Points:
    16
    try running an strace -p PID on the process that sits at the top of the process list.

    See if you get lots of what I am getting which is Operation not Supported.. see Mysql panic post on here.
     
  7. abubin

    abubin Well-Known Member

    Joined:
    Dec 7, 2004
    Messages:
    393
    Likes Received:
    1
    Trophy Points:
    18
    yes, with strace, I do see "Operation not supported". Where is the post to solve this?? Can't find it with search "mysql panic".

    edit: nvm, found the link here : http://forums.cpanel.net/showthread.php?t=49860
     
    #7 abubin, Feb 17, 2006
    Last edited: Feb 17, 2006
Loading...

Share This Page