regular server crash - any ideas...

[4u123]

Can anyone help me identify why this is running out of memory ?

Message log shows...

Feb 11 07:41:11 SERVERNAME kernel: oom-killer: gfp_mask=0x1d2
Feb 11 07:41:29 SERVERNAME kernel: DMA per-cpu:
Feb 11 07:41:46 SERVERNAME kernel: cpu 0 hot: low 2, high 6, batch 1
Feb 11 07:42:36 SERVERNAME kernel: cpu 0 cold: low 0, high 2, batch 1
Feb 11 07:42:47 SERVERNAME kernel: Normal per-cpu:
Feb 11 07:43:17 SERVERNAME kernel: cpu 0 hot: low 32, high 96, batch 16
Feb 11 07:43:41 SERVERNAME kernel: cpu 0 cold: low 0, high 32, batch 16
Feb 11 07:43:49 SERVERNAME kernel: HighMem per-cpu: empty
Feb 11 07:44:19 SERVERNAME kernel:
Feb 11 07:44:42 SERVERNAME kernel: Free pages: 1636kB (0kB HighMem)
Feb 11 07:45:35 SERVERNAME kernel: Active:251 inactive:110681 dirty:0 writeback:110734 unstable:0 free:409 slab:10564 mapped:4845 pagetables:3367
Feb 11 07:45:38 SERVERNAME kernel: DMA free:20kB min:20kB low:40kB high:60kB active:0kB inactive:11124kB present:16384kB
Feb 11 07:45:44 SERVERNAME kernel: protections[]: 0 0 0
Feb 11 07:45:45 SERVERNAME kernel: Normal free:1616kB min:692kB low:1384kB high:2076kB active:1004kB inactive:431600kB present:499648kB
Feb 11 07:45:46 SERVERNAME kernel: protections[]: 0 0 0
Feb 11 07:45:48 SERVERNAME kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB
Feb 11 07:45:50 SERVERNAME kernel: protections[]: 0 0 0
Feb 11 07:45:50 SERVERNAME kernel: DMA: 1*4kB 0*8kB 1*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 20kB
Feb 11 07:45:50 SERVERNAME kernel: Normal: 166*4kB 83*8kB 0*16kB 3*32kB 3*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1616kB
Feb 11 07:45:50 SERVERNAME kernel: HighMem: empty
Feb 11 07:46:09 SERVERNAME kernel: Swap cache: add 831251, delete 720514, find 6713034/6771598, race 0+9
Feb 11 07:47:59 SERVERNAME kernel: Out of Memory: Killed process 4572 (mysqld).
Feb 11 07:48:24 SERVERNAME kernel: Out of Memory: Killed process 4622 (mysqld).
Feb 11 07:48:31 SERVERNAME kernel: Out of Memory: Killed process 4624 (mysqld).
Feb 11 07:48:51 SERVERNAME kernel: Out of Memory: Killed process 4638 (mysqld).
Feb 11 07:48:58 SERVERNAME kernel: Out of Memory: Killed process 4639 (mysqld).
Feb 11 07:49:37 SERVERNAME kernel: Out of Memory: Killed process 4640 (mysqld).
Feb 11 07:49:42 SERVERNAME kernel: Out of Memory: Killed process 4641 (mysqld).
Feb 11 07:49:50 SERVERNAME kernel: Out of Memory: Killed process 4642 (mysqld).
Feb 11 07:50:00 SERVERNAME kernel: Out of Memory: Killed process 4643 (mysqld).
Feb 11 07:50:31 SERVERNAME kernel: Out of Memory: Killed process 4644 (mysqld).
Feb 11 07:52:16 SERVERNAME pure-ftpd: (?@IP.IP.IP.IP) [INFO] New connection from IP.IP.IP.IP
Feb 11 07:54:34 SERVERNAME kernel: Out of Memory: Killed process 4645 (mysqld).
Feb 11 07:55:33 SERVERNAME kernel: Out of Memory: Killed process 4646 (mysqld).
Feb 11 07:57:13 SERVERNAME kernel: Out of Memory: Killed process 4647 (mysqld).
Feb 11 07:57:14 SERVERNAME kernel: Out of Memory: Killed process 4648 (mysqld).
Feb 11 07:57:14 SERVERNAME kernel: Out of Memory: Killed process 4649 (mysqld).
Feb 11 07:57:14 SERVERNAME kernel: Out of Memory: Killed process 4650 (mysqld).
Feb 11 07:57:16 SERVERNAME kernel: Out of Memory: Killed process 4651 (mysqld).
Feb 11 07:57:41 SERVERNAME kernel: Out of Memory: Killed process 4652 (mysqld).
Feb 11 07:57:31 SERVERNAME pure-ftpd: (?@IP.IP.IP.IP) [INFO] Logout.
Feb 11 07:57:43 SERVERNAME kernel: Out of Memory: Killed process 4653 (mysqld).
Feb 11 08:00:35 SERVERNAME kernel: Out of Memory: Killed process 4654 (mysqld).
Feb 11 08:00:39 SERVERNAME kernel: Out of Memory: Killed process 4655 (mysqld).
Feb 11 08:00:39 SERVERNAME freshclam[3639]: Received signal: wake up
Feb 11 08:00:42 SERVERNAME kernel: Out of Memory: Killed process 4656 (mysqld).
Feb 11 08:00:44 SERVERNAME pure-ftpd: (?@IP.IP.IP.IP) [INFO] New connection from IP.IP.IP.IP
Feb 11 08:00:45 SERVERNAME crond(pam_unix)[21809]: session opened for user root by (uid=0)
Feb 11 08:00:45 SERVERNAME crond(pam_unix)[21865]: session opened for user tvacnnet by (uid=0)
Feb 11 08:00:45 SERVERNAME crond(pam_unix)[21859]: session opened for user root by (uid=0)
Feb 11 08:00:46 SERVERNAME kernel: Out of Memory: Killed process 4663 (mysqld).
Feb 11 08:00:47 SERVERNAME freshclam[3639]: ClamAV update process started at Sat Feb 11 08:00:46 2006
Feb 11 08:00:49 SERVERNAME pure-ftpd: (?@IP.IP.IP.IP) [INFO] Logout.
Feb 11 08:01:42 SERVERNAME crond(pam_unix)[22055]: session opened for user root by (uid=0)
Feb 11 08:01:58 SERVERNAME kernel: oom-killer: gfp_mask=0xd2
Feb 11 08:02:05 SERVERNAME crond(pam_unix)[22162]: session opened for user root by (uid=0)
Feb 11 08:02:05 SERVERNAME crond(pam_unix)[22161]: session opened for user root by (uid=0)
Feb 11 08:02:05 SERVERNAME freshclam[3639]: main.cvd is up to date (version: 35, sigs: 41649, f-level: 6, builder: tkojm)
Feb 11 08:02:29 SERVERNAME crond(pam_unix)[21801]: session closed for user root

LOTS MORE CROND SESSIONS HERE SAME AS ABOVE (didnt paste them all)

Feb 11 08:11:51 SERVERNAME pure-ftpd: (?@IP.IP.IP.IP) [INFO] New connection from IP.IP.IP.IP
Feb 11 08:16:25 SERVERNAME crond(pam_unix)[21852]: session closed for user root
Feb 11 08:24:51 SERVERNAME pure-ftpd: (?@IP.IP.IP.IP) [INFO] New connection from IP.IP.IP.IP
Feb 11 08:31:19 SERVERNAME pure-ftpd: (?@IP.IP.IP.IP) [INFO] New connection from IP.IP.IP.IP
Feb 11 08:31:42 SERVERNAME kernel: DMA per-cpu:
Feb 11 08:32:06 SERVERNAME crond(pam_unix)[22417]: session opened for user root by Feb 11 08:32:43 SERVERNAME crond(pam_unix)[21795]: session closed for user root
Feb 11 08:32:46 SERVERNAME crond(pam_unix)[21859]: session closed for user root
Feb 11 08:33:07 SERVERNAME crond(pam_unix)[21837]: session closed for user root
Feb 11 08:33:17 SERVERNAME crond(pam_unix)[21864]: session closed for user root
Feb 11 08:33:20 SERVERNAME crond(pam_unix)[21798]: session closed for user root
Feb 11 08:33:38 SERVERNAME crond(pam_unix)[21857]: session closed for user root
Feb 11 08:33:47 SERVERNAME crond(pam_unix)[21820]: session closed for user root
Feb 11 08:33:46 SERVERNAME crond(pam_unix)[21804]: session closed for user root
Feb 11 08:33:47 SERVERNAME crond(pam_unix)[21789]: session closed for user root
Feb 11 08:33:49 SERVERNAME crond(pam_unix)[21807]: session closed for user root
Feb 11 08:33:49 SERVERNAME crond(pam_unix)[21856]: session closed for user root
Feb 11 08:33:49 SERVERNAME crond(pam_unix)[21822]: session closed for user root
Feb 11 08:33:52 SERVERNAME crond(pam_unix)[21793]: session closed for user root
Feb 11 08:34:16 SERVERNAME crond(pam_unix)[23400]: session opened for user root by (uid=0)
Feb 11 08:34:47 SERVERNAME crond(pam_unix)[21860]: session closed for user root
Feb 11 09:05:28 SERVERNAME syslogd 1.4.1: restart.

Can a mysql process eat up all the memory like this ? Is a user abusing mysql ?

I guess the reason for the number of crond entries is related to the frequency we are running sim and antidos...

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
HOME=/

# run-parts
01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly
*/5 * * * * root /usr/local/sim/sim -q >> /dev/null 2>&1
*/2 * * * * root /etc/apf/ad/antidos -a >> /dev/null 2>&1

Box has only 512MB but this isnt normally a problem.

Any ideas? I'm still not an expert on this yet.

[serversphere]

They might not be purposefully abusing it, but they might be running something that is too intensive for the box. They are probably just running a script that is very sql intensive - check to see what it is that script is doing. 512MB can be used up pretty quick on a mysql intensive box but do a search on 'optimizing mysql' and see if that helps.