Poss memory leak ?

[fuzioneer]

We have centos 3.4 running on our server.

We have started encountering random hangs

Upon investgation it appears that server hangs are caused by slowly building memory usage until eventually server falls over.

My linux guy has been running some scripts to track memory usage and it appears to be MySql that is slowly but surely eating up ram

We have a lot of phpnuke and forums running on this particular server so mysql does get hammered.

relevant snippet from my.cnf:-

# The MySQL server
[mysqld]
port = 3306
socket = /var/lib/mysql/mysql.sock
skip-locking
key_buffer = 384M
max_allowed_packet = 6M
table_cache = 512
sort_buffer_size = 2M
read_buffer_size = 2M
myisam_sort_buffer_size = 64M
thread_cache = 8
query_cache_size = 32M
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8


He wrote a small perl script to ascertain mysql process memory usage as follows:-

#!/usr/bin/perl

$date = scalar localtime;
$ps = `ps -eo pid,user,rss,args`;
@procs = split /\n/, $ps;

open LOGF, ">>/var/tmp/process_mem.log";
print LOGF "\n$date\n";

foreach $proc (@procs) {
chomp $proc;
@a = split /\s+/, $proc;
if (@a[2] >= 71680) {
print LOGF "@a[3] (@a[0]) using more than 70mb memory - (@a[2] kb)\n";
}
}

print LOGF "\n*******


initially the log had entries like this from last night after a reboot:-
Thu Sep 8 23:30:00 2005
/usr/sbin/mysqld (14878) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (15946) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (20824) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (10875) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (13544) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (13607) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (14470) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (15814) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (16617) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (17514) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (19699) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (19858) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (19862) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (21547) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (23329) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (23470) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (23473) using more than 70mb memory - (94584 kb)
/usr/sbin/mysqld (23624) using more than 70mb memory - (94584 kb)


at present time this is already up to the following:-
Fri Sep 9 10:30:00 2005
/usr/sbin/mysqld (20824) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (19858) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (26125) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (11562) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (17485) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (19113) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (19340) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (22260) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (23813) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (25843) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (26635) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (26636) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (26637) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (26638) using more than 70mb memory - (139728 kb)
/usr/sbin/mysqld (26639) using more than 70mb memory - (139728 kb)

and indications are this will just increase over time until next crash.

free command result:-
free
total used free shared buffers cached
Mem: 2045964 1581428 464536 0 82544 442620
-/+ buffers/cache: 1056264 989700
Swap: 2040244 198252 1841992

Below is the MRTG swap file stats for the server

HELP
ANyone got any ideas as to why this is happening ?

[xixi]

Any progress in that case ?
For a few days on my server i also noticed the same problem, mysqld is eating up all avalible memory and after 2-3 hours server is hanging out.
How can i check which database is eatng it up ?

[positive]

i have the same problem.
any solutions?

[abubin]

me too have mysqld eating up memory. I have lot and lots of mysql processes running with each taking like 70mb memory. But for my case, I need to restart mysql after a week or less. But still it's really annoying problem.

Please help...

[abubin]

here is the problem with top screenshot :


12:26:09 up 83 days, 2:09, 1 user, load average: 0.50, 0.29, 0.21
226 processes: 223 sleeping, 1 running, 2 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 14.8% 0.0% 1.9% 0.0% 0.0% 4.4% 78.7%
cpu00 5.5% 0.0% 1.8% 0.0% 0.0% 0.0% 92.5%
cpu01 1.8% 0.0% 1.8% 0.0% 0.0% 0.0% 96.2%
Mem: 1015908k av, 938472k used, 77436k free, 0k shrd, 57860k buff
682480k actv, 139464k in_d, 13656k in_c
Swap: 2048276k av, 176212k used, 1872064k free 361744k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
26798 mysql 15 0 74128 70M 1608 S 0.0 7.1 0:21 1 mysqld
26799 mysql 15 0 74128 70M 1608 S 0.0 7.1 0:02 0 mysqld
26800 mysql 15 0 74128 70M 1608 S 0.0 7.1 0:12 0 mysqld
26801 mysql 15 0 74128 70M 1608 S 0.0 7.1 7:57 0 mysqld
26809 mysql 15 0 74128 70M 1608 S 0.0 7.1 7:57 1 mysqld
26923 mysql 15 0 74128 70M 1608 S 0.0 7.1 8:05 0 mysqld
27098 mysql 15 0 74128 70M 1608 S 0.0 7.1 6:48 0 mysqld
27306 mysql 15 0 74128 70M 1608 S 0.0 7.1 7:50 1 mysqld
27321 mysql 15 0 74128 70M 1608 S 0.0 7.1 6:55 1 mysqld
29520 mysql 15 0 74128 70M 1608 S 0.0 7.1 7:37 1 mysqld
8313 mysql 15 0 74128 70M 1608 S 0.0 7.1 6:31 0 mysqld
22313 mysql 15 0 74128 70M 1608 S 0.0 7.1 6:12 0 mysqld
22314 mysql 15 0 74128 70M 1608 S 0.0 7.1 6:19 1 mysqld
9380 mysql 15 0 74128 70M 1608 S 0.0 7.1 4:59 1 mysqld
9381 mysql 15 0 74128 70M 1608 S 0.0 7.1 5:10 1 mysqld
9382 mysql 15 0 74128 70M 1608 S 0.0 7.1 5:19 0 mysqld
9383 mysql 15 0 74128 70M 1608 S 0.0 7.1 5:00 0 mysqld
9384 mysql 15 0 74128 70M 1608 S 0.0 7.1 4:47 0 mysqld
9386 mysql 15 0 74128 70M 1608 S 0.0 7.1 4:45 0 mysqld
9387 mysql 15 0 74128 70M 1608 S 0.0 7.1 4:54 1 mysqld
9388 mysql 16 0 74128 70M 1608 S 0.0 7.1 5:04 0 mysqld
9396 mysql 15 0 74128 70M 1608 S 0.0 7.1 5:46 1 mysqld
19105 mysql 15 0 74128 70M 1608 S 0.0 7.1 3:39 1 mysqld
19106 mysql 15 0 74128 70M 1608 S 0.0 7.1 3:24 0 mysqld
19107 mysql 15 0 74128 70M 1608 S 0.0 7.1 2:46 0 mysqld


This is really crazy....anyone have any idea what is happening?

[WestBend]

try running an strace -p PID on the process that sits at the top of the process list.

See if you get lots of what I am getting which is Operation not Supported.. see Mysql panic post on here.

[abubin]

try running an strace -p PID on the process that sits at the top of the process list.

See if you get lots of what I am getting which is Operation not Supported.. see Mysql panic post on here.

yes, with strace, I do see "Operation not supported". Where is the post to solve this?? Can't find it with search "mysql panic".

edit: nvm, found the link here : http://forums.cpanel.net/showthread.php?t=49860