The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

cpsrvd.so segfault and recurring page allocation failures

Discussion in 'General Discussion' started by s2s, Dec 30, 2013.

  1. s2s

    s2s Member

    Joined:
    Nov 21, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Hey guys, so I just came back from Christmas holidays and got to reading through the server logs.

    The past few log reports (emailed) contain warnings about page allocation failures:
    - kernel: swapper: page allocation failure. order:1, mode:0x20
    - kernel: kswapd0: page allocation failure. order:1, mode:0x20
    - kernel: imap: page allocation failure. order:1, mode:0x20

    Checking previous logs this seems to stem from a cpsrvd issue (botched update causing out of memory??). The VPS has been running fine and the website is still up, which is kind of why I'm only noticing this now as I haven't had any warnings about server downtime, etc.

    The VPS doesn't seem to be under any heavy load or running out of resources:

    CPU Load: average 0.10
    MEM: 26.3% (3.74G total)
    SWAP: 3.0% (1024M total)

    Following are logs for each incident:

    cpsrvd problem (2013-12-22 00:44:01)
    Code:
    [12/22/2013:00:44:01 -0000] info [cpsrvd] reloading config based on -HUP signal
    [2013-12-22 00:44:22 +0000] warn [cpsessetup] Failed to create: Cpanel::Cpses::Setup::Postgres: could not determine postgres user at /usr/local/cpanel/Cpanel/Cpses/Setup/Postgres.pm line 21.
     at /usr/local/cpanel/bin/cpsessetup line 99
    	bin::cpsessetup::process_cpses_setup_modules() called at /usr/local/cpanel/bin/cpsessetup line 34
    	bin::cpsessetup::script('bin::cpsessetup') called at /usr/local/cpanel/bin/cpsessetup line 24
    [2013-12-22 00:44:34 +0000] die [realadduser] User cpanel already exists
    [2013-12-22 00:44:34 +0000] die [realadduser] User ftp already exists
    This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
    	 "/usr/local/cpanel/base/frontend/x3/branding/the_beach/heading_sprites_bg_snap_to_smallest_width.png".
    This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
    	 "/usr/local/cpanel/base/frontend/x3/branding/motor_city/ui_sprites_bg_snap_to_smallest_width.png".
    This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
    	 "/usr/local/cpanel/base/frontend/x3/branding/crimson_smoke/heading_sprites_bg_snap_to_smallest_width.png".
    This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
    	 "/usr/local/cpanel/base/frontend/x3mail/branding/the_beach/heading_sprites_bg_snap_to_smallest_width.png".
    This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
    	 "/usr/local/cpanel/base/frontend/x3mail/branding/motor_city/ui_sprites_bg_snap_to_smallest_width.png".
    This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
    	 "/usr/local/cpanel/base/frontend/x3mail/branding/crimson_smoke/heading_sprites_bg_snap_to_smallest_width.png".
    [2013-12-22 01:00:56 +0000] warn [whostmgr2] Failed to locate postgresql CLI application
    Restarting cpsrvd daemon process 2011 via /usr/local/cpanel/cpsrvd-ssl
    ==> cpsrvd 11.40.1.7 started
    ==> cpsrvd: preloaded ssl_write_all!
    ==> cpsrvd: loading security policy....Done
    ==> cpsrvd: Setting up native SSL support ... Done
    ==> cpsrvd: transfered port bindings
    ==> cpsrvd: bound to ports
    [B]Out of memory![/B]
    From then on (2013-12-22 00:44:01) I initially had the segfault with cpsrvd.so, then numerous page allocation failrues:

    cpsrvd.so segfault (2013-12-23 03:52:21)
    Code:
    Dec 23 03:52:21 442248 kernel: cpsrvd.so[21715]: segfault at 0 ip 000000361248eff2 sp 00007fff8e325e50 error 4 in libperl.so[3612400000+cf000]
    Dec 23 03:52:27 442248 abrtd: Directory 'ccpp-2013-12-23-03:52:21-21715' creation detected
    Dec 23 03:52:27 442248 abrt[21724]: Saved core dump of pid 21715 (/usr/local/cpanel/perl/Cpanel/cpsrvd.so) to /var/spool/abrt/ccpp-2013-12-23-03:52:21-21715 (483356672 bytes)
    Dec 23 03:52:28 442248 abrtd: Executable '/usr/local/cpanel/perl/Cpanel/cpsrvd.so' doesn't belong to any package and ProcessUnpackaged is set to 'no'
    Dec 23 03:52:28 442248 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2013-12-23-03:52:21-21715' exited with 1
    Dec 23 03:52:28 442248 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2013-12-23-03:52:21-21715'
    page allocation failure: swapper (2013-12-23 12:01:03)
    Code:
    Dec 23 12:01:03 442248 kernel: swapper: page allocation failure. order:1, mode:0x20
    Dec 23 12:01:03 442248 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.14.1.el6.x86_64 #1
    Dec 23 12:01:03 442248 kernel: Call Trace:
    Dec 23 12:01:03 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
    Dec 23 12:01:03 442248 kernel: [<ffffffffa0115800>] ? virtnet_remove+0x11/0x3f [virtio_net]
    Dec 23 12:01:03 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
    Dec 23 12:01:03 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
    Dec 23 12:01:03 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
    Dec 23 12:01:03 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
    Dec 23 12:01:03 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
    Dec 23 12:01:03 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
    Dec 23 12:01:03 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
    Dec 23 12:01:03 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
    Dec 23 12:01:03 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
    Dec 23 12:01:03 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
    Dec 23 12:01:03 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
    Dec 23 12:01:03 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
    Dec 23 12:01:03 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
    Dec 23 12:01:03 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
    Dec 23 12:01:03 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
    Dec 23 12:01:03 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
    Dec 23 12:01:03 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
    Dec 23 12:01:03 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
    Dec 23 12:01:03 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
    Dec 23 12:01:03 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
    Dec 23 12:01:03 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
    Dec 23 12:01:03 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
    Dec 23 12:01:03 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
    Dec 23 12:01:03 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
    Dec 23 12:01:03 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
    Dec 23 12:01:03 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
    Dec 23 12:01:03 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
    Dec 23 12:01:03 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
    Dec 23 12:01:03 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
    Dec 23 12:01:03 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
    Dec 23 12:01:03 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
    Dec 23 12:01:03 442248 kernel: <EOI>  [<ffffffff810387db>] ? native_safe_halt+0xb/0x10
    Dec 23 12:01:03 442248 kernel: [<ffffffff810149cd>] ? default_idle+0x4d/0xb0
    Dec 23 12:01:03 442248 kernel: [<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
    Dec 23 12:01:03 442248 kernel: [<ffffffff814e4baa>] ? rest_init+0x7a/0x80
    Dec 23 12:01:03 442248 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
    Dec 23 12:01:03 442248 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
    Dec 23 12:01:03 442248 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
    page allocation failure: kswapd0 (2013-12-23 20:01:00)
    Code:
    Dec 23 20:01:00 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
    Dec 23 20:01:00 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
    Dec 23 20:01:00 442248 kernel: Call Trace:
    Dec 23 20:01:00 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
    Dec 23 20:01:00 442248 kernel: [<ffffffff8143b600>] ? dev_hard_start_xmit+0x1e0/0x3f0
    Dec 23 20:01:00 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
    Dec 23 20:01:00 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
    Dec 23 20:01:00 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
    Dec 23 20:01:00 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
    Dec 23 20:01:00 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
    Dec 23 20:01:00 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
    Dec 23 20:01:00 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
    Dec 23 20:01:00 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
    Dec 23 20:01:00 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
    Dec 23 20:01:00 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
    Dec 23 20:01:00 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
    Dec 23 20:01:00 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
    Dec 23 20:01:00 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
    Dec 23 20:01:00 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
    Dec 23 20:01:00 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
    Dec 23 20:01:00 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
    Dec 23 20:01:00 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
    Dec 23 20:01:00 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
    Dec 23 20:01:00 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
    Dec 23 20:01:00 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
    Dec 23 20:01:00 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
    Dec 23 20:01:00 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
    Dec 23 20:01:00 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
    Dec 23 20:01:00 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
    Dec 23 20:01:00 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
    Dec 23 20:01:00 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
    Dec 23 20:01:00 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
    Dec 23 20:01:00 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
    Dec 23 20:01:00 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
    Dec 23 20:01:00 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
    Dec 23 20:01:00 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
    Dec 23 20:01:00 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
    Dec 23 20:01:00 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
    Dec 23 20:01:00 442248 kernel: <EOI>  [<ffffffff81500982>] ? _spin_lock+0x12/0x30
    Dec 23 20:01:00 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
    Dec 23 20:01:00 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
    Dec 23 20:01:00 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
    Dec 23 20:01:00 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
    Dec 23 20:01:00 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
    Dec 23 20:01:00 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
    Dec 23 20:01:00 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
    Dec 23 20:01:00 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
    Dec 23 20:01:00 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
    Dec 23 20:01:00 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
    Dec 23 20:01:00 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
    Dec 23 20:01:00 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
    Dec 23 20:01:00 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
    Dec 23 20:01:00 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
    Dec 23 20:01:00 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
    Dec 23 20:01:00 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
    Dec 23 20:01:00 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
    Dec 23 20:01:00 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
    Dec 23 20:01:00 442248 kernel: Call Trace:
    Dec 23 20:01:00 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
    Dec 23 20:01:00 442248 kernel: [<ffffffff8102b400>] ? lapic_next_event+0x10/0x30
    Dec 23 20:01:00 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
    Dec 23 20:01:00 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
    Dec 23 20:01:00 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
    Dec 23 20:01:00 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
    Dec 23 20:01:00 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
    Dec 23 20:01:00 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
    Dec 23 20:01:00 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
    Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
    Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
    Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
    Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
    Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
    Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
    Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
    Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
    Dec 23 20:01:01 442248 kernel: <EOI>  [<ffffffff81500982>] ? _spin_lock+0x12/0x30
    Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
    Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
    Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
    Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
    Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
    Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
    Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
    Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
    Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
    Dec 23 20:01:01 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
    Dec 23 20:01:01 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
    Dec 23 20:01:01 442248 kernel: Call Trace:
    Dec 23 20:01:01 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
    Dec 23 20:01:01 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
    Dec 23 20:01:01 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
    Dec 23 20:01:01 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
    Dec 23 20:01:01 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
    Dec 23 20:01:01 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
    Dec 23 20:01:01 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
    Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
    Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
    Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
    Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
    Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
    Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
    Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
    Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
    Dec 23 20:01:01 442248 kernel: <EOI>  [<ffffffff81500982>] ? _spin_lock+0x12/0x30
    Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
    Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
    Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
    Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
    Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
    Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
    Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
    Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
    Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
    Dec 23 20:01:01 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
    Dec 23 20:01:01 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
    Dec 23 20:01:01 442248 kernel: Call Trace:
    Dec 23 20:01:01 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
    Dec 23 20:01:01 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
    Dec 23 20:01:01 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
    Dec 23 20:01:01 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
    Dec 23 20:01:01 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
    Dec 23 20:01:01 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
    Dec 23 20:01:01 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
    Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
    Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
    Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
    Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
    Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
    Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
    Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
    Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
    Dec 23 20:01:01 442248 kernel: <EOI>  [<ffffffff81500982>] ? _spin_lock+0x12/0x30
    Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
    Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
    Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
    Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
    Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
    Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
    Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
    Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
    Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
    Dec 23 20:01:01 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
    Dec 23 20:01:01 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
    Dec 23 20:01:01 442248 kernel: Call Trace:
    Dec 23 20:01:01 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
    Dec 23 20:01:01 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
    Dec 23 20:01:01 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
    Dec 23 20:01:01 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
    Dec 23 20:01:01 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
    Dec 23 20:01:01 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
    Dec 23 20:01:01 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
    Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
    Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
    Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
    Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
    Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
    Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
    Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
    Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
    Dec 23 20:01:01 442248 kernel: <EOI>  [<ffffffff81500982>] ? _spin_lock+0x12/0x30
    Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
    Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
    Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
    Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
    Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
    Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
    Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
    Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
    Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
    Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
    Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
    Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
    page allocation failure: swapper (2013-12-26 12:00:31)
    Code:
    Dec 26 12:00:31 442248 kernel: swapper: page allocation failure. order:1, mode:0x20
    Dec 26 12:00:31 442248 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.14.1.el6.x86_64 #1
    Dec 26 12:00:31 442248 kernel: Call Trace:
    Dec 26 12:00:31 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
    Dec 26 12:00:31 442248 kernel: [<ffffffff8143b600>] ? dev_hard_start_xmit+0x1e0/0x3f0
    Dec 26 12:00:31 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
    Dec 26 12:00:31 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
    Dec 26 12:00:31 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
    Dec 26 12:00:31 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
    Dec 26 12:00:31 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
    Dec 26 12:00:31 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
    Dec 26 12:00:31 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
    Dec 26 12:00:31 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
    Dec 26 12:00:31 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
    Dec 26 12:00:31 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
    Dec 26 12:00:31 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
    Dec 26 12:00:31 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
    Dec 26 12:00:31 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
    Dec 26 12:00:31 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
    Dec 26 12:00:31 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
    Dec 26 12:00:31 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
    Dec 26 12:00:31 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
    Dec 26 12:00:31 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
    Dec 26 12:00:31 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
    Dec 26 12:00:31 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
    Dec 26 12:00:31 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
    Dec 26 12:00:31 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
    Dec 26 12:00:31 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
    Dec 26 12:00:31 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
    Dec 26 12:00:31 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
    Dec 26 12:00:31 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
    Dec 26 12:00:31 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
    Dec 26 12:00:31 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
    Dec 26 12:00:31 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
    Dec 26 12:00:31 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
    Dec 26 12:00:31 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
    Dec 26 12:00:31 442248 kernel: <EOI>  [<ffffffff810387db>] ? native_safe_halt+0xb/0x10
    Dec 26 12:00:31 442248 kernel: [<ffffffff810149cd>] ? default_idle+0x4d/0xb0
    Dec 26 12:00:31 442248 kernel: [<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
    Dec 26 12:00:31 442248 kernel: [<ffffffff814e4baa>] ? rest_init+0x7a/0x80
    Dec 26 12:00:31 442248 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
    Dec 26 12:00:31 442248 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
    Dec 26 12:00:31 442248 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
    Dec 26 12:00:31 442248 kernel: swapper: page allocation failure. order:1, mode:0x20
    Dec 26 12:00:31 442248 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.14.1.el6.x86_64 #1
    Dec 26 12:00:31 442248 kernel: Call Trace:
    Dec 26 12:00:31 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
    Dec 26 12:00:31 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
    Dec 26 12:00:31 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
    Dec 26 12:00:31 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
    Dec 26 12:00:31 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
    Dec 26 12:00:31 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
    Dec 26 12:00:31 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
    Dec 26 12:00:31 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
    Dec 26 12:00:31 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
    Dec 26 12:00:31 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
    Dec 26 12:00:31 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
    Dec 26 12:00:31 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
    Dec 26 12:00:31 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
    Dec 26 12:00:31 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
    Dec 26 12:00:31 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
    Dec 26 12:00:31 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
    Dec 26 12:00:31 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
    Dec 26 12:00:31 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
    Dec 26 12:00:31 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
    Dec 26 12:00:31 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
    Dec 26 12:00:31 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
    Dec 26 12:00:31 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
    Dec 26 12:00:31 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
    Dec 26 12:00:31 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
    Dec 26 12:00:31 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
    Dec 26 12:00:31 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
    Dec 26 12:00:31 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
    Dec 26 12:00:31 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
    Dec 26 12:00:31 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
    Dec 26 12:00:31 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
    Dec 26 12:00:31 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
    Dec 26 12:00:31 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
    Dec 26 12:00:31 442248 kernel: <EOI>  [<ffffffff810387db>] ? native_safe_halt+0xb/0x10
    Dec 26 12:00:31 442248 kernel: [<ffffffff810149cd>] ? default_idle+0x4d/0xb0
    Dec 26 12:00:31 442248 kernel: [<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
    Dec 26 12:00:31 442248 kernel: [<ffffffff814e4baa>] ? rest_init+0x7a/0x80
    Dec 26 12:00:31 442248 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
    Dec 26 12:00:31 442248 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
    Dec 26 12:00:31 442248 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
    page allocation failure: imap (2013-12-28 00:36:05)
    Code:
    Dec 28 00:36:05 442248 kernel: imap: page allocation failure. order:1, mode:0x20
    Dec 28 00:36:05 442248 kernel: Pid: 6126, comm: imap Not tainted 2.6.32-279.14.1.el6.x86_64 #1
    Dec 28 00:36:05 442248 kernel: Call Trace:
    Dec 28 00:36:05 442248 kernel: [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
    Dec 28 00:36:05 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
    Dec 28 00:36:05 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
    Dec 28 00:36:05 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
    Dec 28 00:36:05 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
    Dec 28 00:36:05 442248 kernel: [<ffffffff8116354a>] ? kmem_cache_alloc_notrace+0xfa/0x130
    Dec 28 00:36:05 442248 kernel: [<ffffffff810cddea>] ? audit_log_d_path+0x5a/0x140
    Dec 28 00:36:05 442248 kernel: [<ffffffff81237373>] ? common_lsm_audit+0x2e3/0x540
    Dec 28 00:36:05 442248 kernel: [<ffffffff8121a4e3>] ? avc_audit+0x73/0xe0
    Dec 28 00:36:05 442248 kernel: [<ffffffff8121ac2f>] ? avc_has_perm_noaudit+0x3df/0x470
    Dec 28 00:36:05 442248 kernel: [<ffffffff81189b1f>] ? do_lookup+0x9f/0x230
    Dec 28 00:36:05 442248 kernel: [<ffffffff8121ad31>] ? avc_has_perm+0x71/0x90
    Dec 28 00:36:05 442248 kernel: [<ffffffff8121c734>] ? inode_has_perm+0x54/0xa0
    Dec 28 00:36:05 442248 kernel: [<ffffffff8119a960>] ? mntput_no_expire+0x30/0x110
    Dec 28 00:36:05 442248 kernel: [<ffffffff81188961>] ? path_put+0x31/0x40
    Dec 28 00:36:05 442248 kernel: [<ffffffff8121cd0b>] ? dentry_has_perm+0x5b/0x80
    Dec 28 00:36:05 442248 kernel: [<ffffffff8121a550>] ? avc_audit_pre_callback+0x0/0x130
    Dec 28 00:36:05 442248 kernel: [<ffffffff8121a680>] ? avc_audit_post_callback+0x0/0x150
    Dec 28 00:36:05 442248 kernel: [<ffffffff8121cdbc>] ? selinux_inode_getattr+0x2c/0x30
    Dec 28 00:36:05 442248 kernel: [<ffffffff812147d3>] ? security_inode_getattr+0x23/0x30
    Dec 28 00:36:05 442248 kernel: [<ffffffff81180a4f>] ? vfs_getattr+0x2f/0x80
    Dec 28 00:36:05 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
    Dec 28 00:36:05 442248 kernel: [<ffffffff81180b00>] ? vfs_fstatat+0x60/0x80
    Dec 28 00:36:05 442248 kernel: [<ffffffff81180c4b>] ? vfs_stat+0x1b/0x20
    Dec 28 00:36:05 442248 kernel: [<ffffffff81180c74>] ? sys_newstat+0x24/0x50
    Dec 28 00:36:05 442248 kernel: [<ffffffff810d6d42>] ? audit_syscall_entry+0x272/0x2a0
    Dec 28 00:36:05 442248 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
    Dec 28 00:36:05 442248 kernel: Mem-Info:
    Dec 28 00:36:05 442248 kernel: Node 0 DMA per-cpu:
    Dec 28 00:36:05 442248 kernel: CPU    0: hi:    0, btch:   1 usd:   0
    Dec 28 00:36:05 442248 kernel: CPU    1: hi:    0, btch:   1 usd:   0
    Dec 28 00:36:05 442248 kernel: Node 0 DMA32 per-cpu:
    Dec 28 00:36:05 442248 kernel: CPU    0: hi:  186, btch:  31 usd: 159
    Dec 28 00:36:05 442248 kernel: CPU    1: hi:  186, btch:  31 usd: 165
    Dec 28 00:36:05 442248 kernel: Node 0 Normal per-cpu:
    Dec 28 00:36:05 442248 kernel: CPU    0: hi:  186, btch:  31 usd:  81
    Dec 28 00:36:05 442248 kernel: CPU    1: hi:  186, btch:  31 usd: 170
    Dec 28 00:36:05 442248 kernel: active_anon:92353 inactive_anon:48818 isolated_anon:0
    Dec 28 00:36:05 442248 kernel: active_file:325354 inactive_file:347522 isolated_file:32
    Dec 28 00:36:05 442248 kernel: unevictable:0 dirty:470 writeback:0 unstable:0
    Dec 28 00:36:05 442248 kernel: free:62119 slab_reclaimable:63385 slab_unreclaimable:21855
    Dec 28 00:36:05 442248 kernel: mapped:6554 shmem:59 pagetables:2847 bounce:0
    Dec 28 00:36:05 442248 kernel: Node 0 DMA free:15732kB min:248kB low:308kB high:372kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
    Dec 28 00:36:05 442248 kernel: lowmem_reserve[]: 0 3512 4017 4017
    Dec 28 00:36:05 442248 kernel: Node 0 DMA32 free:236080kB min:58868kB low:73584kB high:88300kB active_anon:313744kB inactive_anon:96472kB active_file:1204912kB inactive_file:1277968kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3596500kB mlocked:0kB dirty:1748kB writeback:0kB mapped:22168kB shmem:224kB slab_reclaimable:209972kB slab_unreclaimable:34260kB kernel_stack:616kB pagetables:3384kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
    Dec 28 00:36:05 442248 kernel: lowmem_reserve[]: 0 0 505 505
    Dec 28 00:36:05 442248 kernel: Node 0 Normal free:15760kB min:8464kB low:10580kB high:12696kB active_anon:55668kB inactive_anon:98800kB active_file:95192kB inactive_file:95320kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:517120kB mlocked:0kB dirty:132kB writeback:0kB mapped:4048kB shmem:12kB slab_reclaimable:42764kB slab_unreclaimable:53160kB kernel_stack:888kB pagetables:8004kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
    Dec 28 00:36:05 442248 kernel: lowmem_reserve[]: 0 0 0 0
    Dec 28 00:36:05 442248 kernel: Node 0 DMA: 3*4kB 1*8kB 2*16kB 2*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15732kB
    Dec 28 00:36:05 442248 kernel: Node 0 DMA32: 45803*4kB 5940*8kB 832*16kB 41*32kB 10*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 246124kB
    Dec 28 00:36:05 442248 kernel: Node 0 Normal: 3219*4kB 784*8kB 13*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 19356kB
    Dec 28 00:36:05 442248 kernel: 667331 total pagecache pages
    Dec 28 00:36:05 442248 kernel: 2033 pages in swap cache
    Dec 28 00:36:05 442248 kernel: Swap cache stats: add 429836, delete 427803, find 68873295/68917309
    Dec 28 00:36:05 442248 kernel: Free swap  = 1023284kB
    Dec 28 00:36:05 442248 kernel: Total swap = 1048568kB
    Dec 28 00:36:05 442248 kernel: 1048575 pages RAM
    Dec 28 00:36:05 442248 kernel: 67802 pages reserved
    Dec 28 00:36:05 442248 kernel: 272595 pages shared
    Dec 28 00:36:05 442248 kernel: 664710 pages non-shared

    The last packages to be updated were:
    Code:
    Dec 18 00:35:32 442248 yum[3703]: Updated: yum-3.2.29-43.el6.centos.noarch
    Dec 20 00:36:19 442248 yum[6213]: Updated: openssl-1.0.1e-16.el6_5.1.x86_64
    Dec 20 00:36:20 442248 yum[6213]: Updated: openssl-devel-1.0.1e-16.el6_5.1.x86_64
    Dec 20 00:36:20 442248 yum[6213]: Updated: systemtap-runtime-2.3-4.el6_5.x86_64
    Dec 21 00:35:35 442248 yum[23604]: Updated: nss-sysinit-3.15.3-3.el6_5.x86_64
    Dec 21 00:35:36 442248 yum[23604]: Updated: nss-3.15.3-3.el6_5.x86_64
    Dec 21 00:35:36 442248 yum[23604]: Updated: nss-tools-3.15.3-3.el6_5.x86_64
    Dec 21 00:35:39 442248 yum[23604]: Updated: ca-certificates-2013.1.95-65.1.el6_5.noarch
    Dec 21 00:35:39 442248 yum[23604]: Updated: pixman-0.26.2-5.1.el6_5.x86_64
    Dec 21 00:35:42 442248 yum[23604]: Updated: tzdata-2013i-1.el6.noarch
    Dec 22 00:41:51 442248 yum[17595]: Updated: iftop-1.0-0.6.pre2.el6.x86_64
    
    I'm not too sure where to begin looking to fix this. Could it be a problem that stems from a failed cpsrvd update? I am also thinking about hardware memory failures, but wanted to check here first.
     
  2. cPanelMichael

    cPanelMichael Forums Analyst
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    30,854
    Likes Received:
    676
    Trophy Points:
    113
    cPanel Access Level:
    Root Administrator
    Hello :)

    Could you let us know the version of cPanel installed on this system? You can find out the version via:

    Code:
    cat /usr/local/cpanel/version
    Try running the following command manually:

    Code:
    /usr/local/cpanel/perl/Cpanel/cpsrvd.so
    Does it complete successfully, or does it hang and continue to run?

    Thank you.
     
  3. cPJerry

    cPJerry Member
    Staff Member

    Joined:
    May 13, 2013
    Messages:
    14
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    If your version reports as 11.40.1.7, please run upcp. There was a bug in build 7 which caused cpsrvd.so to segfault.

    This is resolved in 11.40.1.8

    You can find the change log at http://docs.cpanel.net/twiki/bin/view/AllDocumentation/ChangeLog/CPanelVersion1140

    The case number is: 86349

    You can also delete any core dumps in /usr/local/cpanel/base.

    You can use this command:

    rm -f /usr/local/cpanel/base/core.*

    IMPORTANT: Do not delete /usr/local/cpanel/base/core, this is not a core dump

    :)
     
  4. s2s

    s2s Member

    Joined:
    Nov 21, 2013
    Messages:
    20
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Root Administrator
    Thanks for getting back guys.

    This issue looks to have resolved itself, the last 2 logs don't contain any issues and my current version is : 11.40.1.8
     
Loading...

Share This Page