The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Second time on a week the server dies.

Discussion in 'General Discussion' started by ServerNet, May 6, 2008.

  1. ServerNet

    ServerNet Member

    Joined:
    Aug 19, 2004
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    1
    Hi,

    This is the second time this week that the server completly dies for several hours.
    This is what I found in my /var/logs/messages (Sorry for the long log).
    We totaly lose all contact with the server remotly, we can't reach ssh or anything(domain nor ip).

    Is there anyone who could perhaps explane what's doing on? :confused:

    Best regards

    Code:
    May  5 21:59:18 web1 named[25404]: client 77.91.224.24#39899: view external: error sending response: host unreachable
    May  5 21:59:18 web1 named[25404]: client 77.91.224.24#39899: view external: error sending response: host unreachable
    May  5 21:59:18 web1 kernel: DMA free:1820kB min:96kB low:120kB high:144kB active:2956kB inactive:5076kB present:16384kB pages_scanned:162702 all_unreclaimable? yes
    May  5 21:59:18 web1 named[25404]: client 77.91.224.24#39899: view external: error sending response: host unreachable
    May  5 21:59:18 web1 kernel: lowmem_reserve[]: 0 0 431 431
    May  5 21:59:18 web1 kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
    May  5 21:59:18 web1 kernel: lowmem_reserve[]: 0 0 431 431
    May  5 21:59:18 web1 kernel: Normal free:2600kB min:2608kB low:3260kB high:3912kB active:220344kB inactive:131072kB present:442304kB pages_scanned:2464703 all_unreclaimable? yes
    May  5 21:59:18 web1 kernel: lowmem_reserve[]: 0 0 0 0
    May  5 21:59:18 web1 named[25404]: client 77.91.224.24#39899: view external: error sending response: host unreachable
    May  5 21:59:18 web1 kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
    May  5 21:59:18 web1 kernel: lowmem_reserve[]: 0 0 0 0
    May  5 21:59:18 web1 kernel: DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1820kB
    May  5 21:59:18 web1 kernel: DMA32: empty
    May  5 21:59:18 web1 kernel: Normal: 18*4kB 6*8kB 13*16kB 9*32kB 3*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2600kB
    May  5 21:59:18 web1 kernel: HighMem: empty
    May  5 21:59:18 web1 kernel: Swap cache: add 76579117, delete 76579083, find 1233338680/1244511431, race 1228+75814
    May  5 21:59:18 web1 kernel: Free swap  = 0kB
    May  5 21:59:18 web1 kernel: Total swap = 2048276kB
    May  5 21:59:18 web1 kernel: Free swap:            0kB
    May  5 21:59:18 web1 kernel: 114672 pages of RAM
    May  5 21:59:18 web1 kernel: 0 pages of HIGHMEM
    May  5 21:59:18 web1 kernel: 2070 reserved pages
    May  5 21:59:18 web1 kernel: 5764 pages shared
    May  5 21:59:18 web1 kernel: 35 pages swap cached
    May  5 21:59:18 web1 kernel: 0 pages dirty
    May  5 21:59:18 web1 kernel: 0 pages writeback
    May  5 21:59:18 web1 kernel: 50 pages mapped
    May  5 21:59:18 web1 kernel: 8499 pages slab
    May  5 21:59:18 web1 kernel: 6538 pages pagetables
    May  5 21:59:18 web1 kernel: printk: 43518 messages suppressed.
    May  5 21:59:18 web1 kernel: mysqld invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0
    May  5 21:59:18 web1 kernel:  [<c0452c4e>] out_of_memory+0x3b/0x179
    May  5 21:59:18 web1 kernel:  [<c0454081>] __alloc_pages+0x1fe/0x27e
    May  5 21:59:18 web1 kernel:  [<c0460b69>] read_swap_cache_async+0x2f/0xac
    May  5 21:59:18 web1 kernel:  [<c0459783>] swapin_readahead+0x3a/0x58
    May  5 21:59:18 web1 kernel:  [<c045ad28>] __handle_mm_fault+0x5bc/0x87b
    May  5 21:59:18 web1 kernel:  [<c04dfa31>] copy_to_user+0x31/0x48
    May  5 21:59:18 web1 kernel:  [<c05f549d>] unix_write_space+0xf/0x69
    May  5 21:59:18 web1 kernel:  [<c05fd48f>] do_page_fault+0x20a/0x4b8
    May  5 21:59:18 web1 kernel:  [<c05fd285>] do_page_fault+0x0/0x4b8
    May  5 21:59:18 web1 kernel:  [<c0404a71>] error_code+0x39/0x40
    May  5 21:59:18 web1 kernel:  [<c04df321>] __get_user_4+0x11/0x17
    May  5 21:59:18 web1 kernel:  [<c04397d0>] exit_robust_list+0x16/0xb6
    May  5 21:59:18 web1 kernel:  [<c0437789>] down_read+0x8/0x11
    May  5 21:59:18 web1 kernel:  [<c044ccdd>] taskstats_exit_alloc+0x59/0x69
    May  5 21:59:18 web1 kernel:  [<c0426064>] do_exit+0x1b3/0x746
    May  5 21:59:18 web1 kernel:  [<c042666d>] sys_exit_group+0x0/0xd
    May  5 21:59:18 web1 kernel:  [<c042eeb8>] get_signal_to_deliver+0x375/0x39c
    May  5 21:59:18 web1 kernel:  [<c040356b>] do_notify_resume+0x77/0x67d
    May  5 21:59:18 web1 kernel:  [<c041dc4d>] default_wake_function+0x0/0xc
    May  5 21:59:18 web1 kernel:  [<c0449262>] audit_syscall_exit+0x285/0x2a1
    May  5 21:59:18 web1 kernel:  [<c0403f8e>] work_notifysig+0x13/0x19
    May  5 21:59:18 web1 kernel:  =======================
    May  5 21:59:18 web1 kernel: Mem-info:
    May  5 21:59:18 web1 kernel: DMA per-cpu:
    May  5 21:59:18 web1 kernel: cpu 0 hot: high 0, batch 1 used:0
    May  5 21:59:18 web1 kernel: cpu 0 cold: high 0, batch 1 used:0
    May  5 21:59:18 web1 kernel: cpu 1 hot: high 0, batch 1 used:0
    May  5 21:59:18 web1 kernel: cpu 1 cold: high 0, batch 1 used:0
    May  5 21:59:18 web1 kernel: DMA32 per-cpu: empty
    May  5 21:59:18 web1 kernel: Normal per-cpu:
    May  5 21:59:18 web1 kernel: cpu 0 hot: high 186, batch 31 used:18
    May  5 21:59:18 web1 kernel: cpu 0 cold: high 62, batch 15 used:31
    May  5 21:59:18 web1 kernel: cpu 1 hot: high 186, batch 31 used:65
    May  5 21:59:18 web1 kernel: cpu 1 cold: high 62, batch 15 used:30
    May  5 21:59:18 web1 kernel: HighMem per-cpu: empty
    May  5 21:59:18 web1 kernel: Free pages:        4420kB (0kB HighMem)
    May  5 21:59:18 web1 kernel: Active:55893 inactive:34090 dirty:0 writeback:0 unstable:0 free:1105 slab:8459 mapped:50 pagetables:6538
    May  5 21:59:18 web1 kernel: DMA free:1820kB min:96kB low:120kB high:144kB active:2564kB inactive:5088kB present:16384kB pages_scanned:135114 all_unreclaimable? yes
    May  5 21:59:18 web1 kernel: lowmem_reserve[]: 0 0 431 431
    May  5 21:59:18 web1 kernel: DMA32 free:0kB min:0kB low:0kB high:0kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
    May  5 21:59:18 web1 kernel: lowmem_reserve[]: 0 0 431 431
    May  5 21:59:18 web1 kernel: Normal free:2600kB min:2608kB low:3260kB high:3912kB active:221264kB inactive:131016kB present:442304kB pages_scanned:4256736 all_unreclaimable? yes
    May  5 21:59:18 web1 kernel: lowmem_reserve[]: 0 0 0 0
    May  5 21:59:18 web1 kernel: HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
    May  5 21:59:18 web1 kernel: lowmem_reserve[]: 0 0 0 0
    May  5 21:59:18 web1 kernel: DMA: 1*4kB 1*8kB 1*16kB 0*32kB 0*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 1820kB
    May  5 21:59:18 web1 kernel: DMA32: empty
    May  5 21:59:18 web1 kernel: Normal: 18*4kB 6*8kB 13*16kB 9*32kB 3*64kB 0*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2600kB
    May  5 21:59:18 web1 kernel: HighMem: empty
    May  5 21:59:18 web1 kernel: Swap cache: add 76579118, delete 76579083, find 1233338680/1244511431, race 1228+75814
    May  5 21:59:18 web1 kernel: Free swap  = 0kB
    May  5 21:59:18 web1 kernel: Total swap = 2048276kB
    May  5 21:59:18 web1 kernel: Free swap:            0kB
    May  5 21:59:18 web1 kernel: 114672 pages of RAM
    May  5 21:59:18 web1 kernel: 0 pages of HIGHMEM
    May  5 21:59:18 web1 kernel: 2070 reserved pages
    May  5 21:59:18 web1 kernel: 5639 pages shared
    May  5 21:59:18 web1 kernel: 36 pages swap cached
    May  5 21:59:18 web1 kernel: 0 pages dirty
    May  5 21:59:19 web1 kernel: 0 pages writeback
    May  5 21:59:19 web1 kernel: 50 pages mapped
    May  5 21:59:20 web1 kernel: 8459 pages slab
    May  5 21:59:21 web1 kernel: 6538 pages pagetables
    May  5 21:59:21 web1 kernel: printk: 42245 messages suppressed.
    May  5 21:59:22 web1 kernel: mysqld invoked oom-killer: gfp_mask=0x200d2, order=0, oomkilladj=0
    May  5 21:59:24 web1 kernel:  [<c0452c4e>] out_of_memory+0x3b/0x179
    May  5 21:59:24 web1 kernel:  [<c0454081>] __alloc_pages+0x1fe/0x27e
    May  5 21:59:24 web1 kernel:  [<c0460b69>] read_swap_cache_async+0x2f/0xac
    May  5 21:59:24 web1 kernel:  [<c0459783>] swapin_readahead+0x3a/0x58
    May  5 21:59:26 web1 kernel:  [<c045ad28>] __handle_mm_fault+0x5bc/0x87b
    May  5 21:59:27 web1 kernel:  [<c05fd48f>] do_page_fault+0x20a/0x4b8
    May  5 21:59:27 web1 kernel:  [<c05fd285>] do_page_fault+0x0/0x4b8
    May  5 21:59:28 web1 kernel:  [<c0404a71>] error_code+0x39/0x40
    May  5 21:59:28 web1 kernel:  [<c04df321>] __get_user_4+0x11/0x17
    May  5 21:59:29 web1 kernel:  [<c04397d0>] exit_robust_list+0x16/0xb6
    May  5 21:59:30 web1 kernel:  [<c0437789>] down_read+0x8/0x11
    May  5 21:59:30 web1 kernel:  [<c044ccdd>] taskstats_exit_alloc+0x59/0x69
    May  5 21:59:31 web1 kernel:  [<c0426064>] do_exit+0x1b3/0x746
    May  5 21:59:31 web1 kernel:  [<c042666d>] sys_exit_group+0x0/0xd
    May  5 21:59:31 web1 kernel:  [<c042eeb8>] get_signal_to_deliver+0x375/0x39c
    May  5 21:59:31 web1 kernel:  [<c040356b>] do_notify_resume+0x77/0x67d
    May  5 21:59:32 web1 kernel:  [<c041dc4d>] default_wake_function+0x0/0xc
    May  5 21:59:32 web1 kernel:  [<c05facb9>] schedule+0x90d/0x9ba
    May  5 21:59:32 web1 kernel:  [<c0449262>] audit_syscall_exit+0x285/0x2a1
    May  5 21:59:33 web1 kernel:  [<c0403f8e>] work_notifysig+0x13/0x19
    May  5 21:59:33 web1 kernel:  =======================
    
    
    //Another 40 of those
    
    May  5 21:59:49 web1 kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:13:d3:60:23:76:00:04:23:ca:a7:09:08:00 SRC=218.64.237.219 DST=83.168.247.10 LEN=404 TOS=0x08 PREC=0x20 TTL=118 ID=44208 PROTO=UDP SPT=1273 DPT=1434 LEN=384 
    May  5 21:59:49 web1 kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:13:d3:60:23:76:00:04:23:ca:a7:09:08:00 SRC=59.63.157.142 DST=83.168.247.13 LEN=404 TOS=0x08 PREC=0x20 TTL=53 ID=26517 PROTO=UDP SPT=1341 DPT=1434 LEN=384 
    -||-
    And another 600 blocks within 2 seconds
    -||-
    
    May  5 22:11:27 web1 proftpd[6343]: web1.servernet.se (72.51.35.200[72.51.35.200]) - FTP session opened. 
    May  5 22:11:27 web1 proftpd[6343]: web1.servernet.se (72.51.35.200[72.51.35.200]) - FTP session closed.
    
     
  2. LiNUxG0d

    LiNUxG0d Well-Known Member

    Joined:
    Jun 25, 2003
    Messages:
    206
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Gatineau, Quebec, Canada
    OOM: Out Of Memory.

    Basically, MySQL is probably overloading, large queries, bad queries hanging queries... DoS on the service maybe?

    Try these commands in a shell (SSH) as root:

    Code:
    [root@server ~]# [B][COLOR="Blue"]mysql[/COLOR][/B]
    Welcome to the MySQL monitor.  Commands end with ; or \g.
    Your MySQL connection id is 117741 to server version: 4.1.22-standard
    
    Type 'help;' or '\h' for help. Type '\c' to clear the buffer.
    
    mysql> [B][COLOR="Blue"]show processlist;[/COLOR][/B]
    +--------+------+-----------+------+---------+------+-------+------------------+
    | Id     | User | Host      | db   | Command | Time | State | Info             |
    +--------+------+-----------+------+---------+------+-------+------------------+
    | 117741 | root | localhost | NULL | Query   |    0 | NULL  | show processlist |
    +--------+------+-----------+------+---------+------+-------+------------------+
    1 row in set (0.00 sec)
    
    mysql>
    
    So you would issue the `mysql` command to bring the prompt and you would use the `show processlist;` command to view what's going on with the MySQL daemon.

    Let me know if this helps or not, or any follow-ups you might need. :)

    Regards,
     
Loading...

Share This Page