The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

High server load / out of memory

Discussion in 'General Discussion' started by grantw, May 8, 2015.

  1. grantw

    grantw Registered

    Joined:
    Nov 21, 2012
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Hi all, I'm hoping someone can point me in the right direction with this as I'm no expert when it comes to servers!

    My server has been down/unresponsive twice in the past 2 days, this has just started happening and I've made no changes to the server. After it's occurred I'm left with SSH access and a server that has pretty much no processes running on it.

    Looking into it as far as I'm able, I'm seeing a lot of these entries in var/log/messages, anything between 5 and 20 of these per second before the issue occurs:

    Code:
    May  7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=111.250.188.187 DST=69.10.138.117 LEN=67 TOS=0x00 PREC=0x00 TTL=238 ID=47804 DF PROTO=UDP SPT=39601 DPT=53 LEN=47
    
    Then at some point the following occurs and the server starts shutting processes down until almost nothing is running, the log file then ends until a reboot is done:

    Code:
    May  7 17:48:22 server kernel: lfd - (child) c invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
    
    I'm trying to figure out where to go from here. Is it a DDos, does some firewall config need changing or is there some other process causing this somewhere. In short, I'm lost and any advice to diagnose this further would be greatly appreciated.

    I've included a bigger snippet of that log and also the sar log from the time of the issue where you can see the server load go way high:

    Code:
    03:30:01 PM         1       185      0.00      0.11      0.15
    03:40:01 PM         1       186      0.04      0.03      0.08
    03:50:01 PM         1       198      0.85      0.39      0.20
    04:00:01 PM         1       207      0.31      0.52      0.35
    04:10:01 PM         0       200      0.26      0.42      0.48
    04:20:01 PM         1       192      0.02      0.21      0.35
    04:30:01 PM         6       226      0.27      0.44      0.37
    04:40:02 PM         0       207      1.23      2.28      1.44
    04:50:01 PM        26       267      5.44      3.55      2.33
    05:00:01 PM         0       276      0.13      1.54      2.18
    05:10:01 PM         0       204      0.82      1.24      1.72
    05:20:01 PM         1       214      0.14      0.56      1.17
    05:30:02 PM         1       235      2.80      2.04      1.63
    05:40:05 PM        25       265     17.26      8.98      4.55
    05:50:17 PM         1       382     79.55     34.77     15.53
    Average:            1       194      1.16      0.70      0.48
    
    06:23:46 PM       LINUX RESTART
    
    06:30:01 PM   runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
    06:40:01 PM         1       190      1.00      1.00      0.82
    
    Code:
    May  7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=111.250.188.187 DST=69.10.138.117 LEN=67 TOS=0x00 PREC=0x00 TTL=238 ID=47804 DF PROTO=UDP SPT=39601 DPT=53 LEN=47
    May  7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=37.119.206.219 DST=69.10.138.117 LEN=71 TOS=0x00 PREC=0x00 TTL=244 ID=56014 DF PROTO=UDP SPT=6091 DPT=53 LEN=51
    May  7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=94.115.87.90 DST=69.10.138.116 LEN=69 TOS=0x00 PREC=0x00 TTL=241 ID=22871 DF PROTO=UDP SPT=4674 DPT=53 LEN=49
    May  7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=64.106.94.250 DST=69.10.138.117 LEN=61 TOS=0x00 PREC=0x00 TTL=241 ID=63838 DF PROTO=UDP SPT=56782 DPT=53 LEN=41
    May  7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=98.33.204.243 DST=69.10.138.116 LEN=69 TOS=0x00 PREC=0x00 TTL=239 ID=62156 DF PROTO=UDP SPT=36663 DPT=53 LEN=49
    May  7 17:48:14 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=8.11.53.149 DST=69.10.138.116 LEN=61 TOS=0x00 PREC=0x00 TTL=245 ID=37941 DF PROTO=UDP SPT=17323 DPT=53 LEN=41
    May  7 17:48:14 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=58.17.253.7 DST=69.10.138.117 LEN=73 TOS=0x00 PREC=0x00 TTL=243 ID=1789 DF PROTO=UDP SPT=28459 DPT=53 LEN=53
    May  7 17:48:14 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=63.123.136.49 DST=69.10.138.116 LEN=73 TOS=0x00 PREC=0x00 TTL=244 ID=12424 DF PROTO=UDP SPT=1321 DPT=53 LEN=53
    May  7 17:48:14 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=62.91.116.38 DST=69.10.138.116 LEN=73 TOS=0x00 PREC=0x00 TTL=243 ID=9588 DF PROTO=UDP SPT=61148 DPT=53 LEN=53
    May  7 17:48:14 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=67.105.254.185 DST=69.10.138.117 LEN=65 TOS=0x00 PREC=0x00 TTL=245 ID=47358 DF PROTO=UDP SPT=33677 DPT=53 LEN=45
    May  7 17:48:16 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=110.180.50.152 DST=69.10.138.117 LEN=63 TOS=0x00 PREC=0x00 TTL=244 ID=38706 DF PROTO=UDP SPT=3586 DPT=53 LEN=43
    May  7 17:48:22 server kernel: lfd - (child) c invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
    May  7 17:48:22 server kernel: lfd - (child) c cpuset=/ mems_allowed=0
    May  7 17:48:22 server kernel: Pid: 53710, comm: lfd - (child) c Not tainted 2.6.32-358.14.1.el6.x86_64 #1
    May  7 17:48:22 server kernel: Call Trace:
    May  7 17:48:22 server kernel: [<ffffffff810cb561>] ? cpuset_print_task_mems_allowed+0x91/0xb0
    May  7 17:48:22 server kernel: [<ffffffff8111cd80>] ? dump_header+0x90/0x1b0
    May  7 17:48:22 server kernel: [<ffffffff810e923e>] ? __delayacct_freepages_end+0x2e/0x30
    May  7 17:48:22 server kernel: [<ffffffff8121d2ac>] ? security_real_capable_noaudit+0x3c/0x70
    May  7 17:48:22 server kernel: [<ffffffff8111d202>] ? oom_kill_process+0x82/0x2a0
    May  7 17:48:22 server kernel: [<ffffffff8111d141>] ? select_bad_process+0xe1/0x120
    May  7 17:48:22 server kernel: [<ffffffff8111d640>] ? out_of_memory+0x220/0x3c0
    May  7 17:48:22 server kernel: [<ffffffff8112c2ec>] ? __alloc_pages_nodemask+0x8ac/0x8d0
    May  7 17:48:22 server kernel: [<ffffffff81160aea>] ? alloc_pages_vma+0x9a/0x150
    May  7 17:48:22 server kernel: [<ffffffff811428ad>] ? do_wp_page+0xfd/0x920
    May  7 17:48:22 server kernel: [<ffffffff81155593>] ? swap_info_get+0x63/0xe0
    May  7 17:48:22 server kernel: [<ffffffff811438cd>] ? handle_pte_fault+0x2cd/0xb50
    May  7 17:48:22 server kernel: [<ffffffff8114438a>] ? handle_mm_fault+0x23a/0x310
    May  7 17:48:22 server kernel: [<ffffffff810474e9>] ? __do_page_fault+0x139/0x480
    May  7 17:48:22 server kernel: [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
    May  7 17:48:22 server kernel: [<ffffffff8150dd80>] ? thread_return+0x4e/0x76e
    May  7 17:48:22 server kernel: [<ffffffff81085191>] ? do_sigaction+0x91/0x1d0
    May  7 17:48:22 server kernel: [<ffffffff81188983>] ? set_task_comm+0x53/0x70
    May  7 17:48:22 server kernel: [<ffffffff815137be>] ? do_page_fault+0x3e/0xa0
    May  7 17:48:22 server kernel: [<ffffffff81510b75>] ? page_fault+0x25/0x30
    May  7 17:48:22 server kernel: Mem-Info:
    May  7 17:48:22 server kernel: Node 0 DMA per-cpu:
    May  7 17:48:22 server kernel: CPU    0: hi:    0, btch:   1 usd:   0
    May  7 17:48:22 server kernel: Node 0 DMA32 per-cpu:
    May  7 17:48:22 server kernel: CPU    0: hi:  186, btch:  31 usd:  30
    May  7 17:48:22 server kernel: active_anon:97981 inactive_anon:97705 isolated_anon:864
    May  7 17:48:22 server kernel: active_file:171 inactive_file:317 isolated_file:0
    May  7 17:48:22 server kernel: unevictable:0 dirty:18 writeback:167 unstable:0
    May  7 17:48:22 server kernel: free:12232 slab_reclaimable:4130 slab_unreclaimable:13454
    May  7 17:48:22 server kernel: mapped:232 shmem:17 pagetables:15743 bounce:0
    May  7 17:48:22 server kernel: Node 0 DMA free:4644kB min:668kB low:832kB high:1000kB active_anon:5200kB inactive_anon:5144kB active_file:12kB inactive_file:40kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15368kB mlocked:0kB dirty:0kB writeback:0kB mapped:24kB shmem:0kB slab_reclaimable:48kB slab_unreclaimable:12kB kernel_stack:16kB pagetables:296kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:29 all_unreclaimable? no
    May  7 17:48:22 server kernel: lowmem_reserve[]: 0 993 993 993
    May  7 17:48:22 server kernel: Node 0 DMA32 free:44284kB min:44384kB low:55480kB high:66576kB active_anon:386724kB inactive_anon:385676kB active_file:672kB inactive_file:1228kB unevictable:0kB isolated(anon):3456kB isolated(file):0kB present:1017504kB mlocked:0kB dirty:72kB writeback:668kB mapped:904kB shmem:68kB slab_reclaimable:16472kB slab_unreclaimable:53804kB kernel_stack:3272kB pagetables:62676kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:704 all_unreclaimable? no
    May  7 17:48:22 server kernel: lowmem_reserve[]: 0 0 0 0
    May  7 17:48:22 server kernel: Node 0 DMA: 15*4kB 13*8kB 20*16kB 14*32kB 10*64kB 0*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 4644kB
    May  7 17:48:22 server kernel: Node 0 DMA32: 1175*4kB 488*8kB 396*16kB 633*32kB 70*64kB 6*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 44284kB
    May  7 17:48:22 server kernel: 11419 total pagecache pages
    May  7 17:48:22 server kernel: 10747 pages in swap cache
    May  7 17:48:22 server kernel: Swap cache stats: add 3451250, delete 3440503, find 819493753/820108390
    May  7 17:48:22 server kernel: Free swap  = 0kB
    May  7 17:48:22 server kernel: Total swap = 524280kB
    May  7 17:48:22 server kernel: 262143 pages RAM
    May  7 17:48:22 server kernel: 9171 pages reserved
    May  7 17:48:22 server kernel: 204471 pages shared
    May  7 17:48:22 server kernel: 226961 pages non-shared
    
    Many thanks!
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,678
    Likes Received:
    648
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
Loading...

Share This Page