cpsrvd.so segfault and recurring page allocation failures

S

s2s

Member
Nov 21, 2013
20
1
1
cPanel Access Level
Root Administrator
Hey guys, so I just came back from Christmas holidays and got to reading through the server logs.

The past few log reports (emailed) contain warnings about page allocation failures:
- kernel: swapper: page allocation failure. order:1, mode:0x20
- kernel: kswapd0: page allocation failure. order:1, mode:0x20
- kernel: imap: page allocation failure. order:1, mode:0x20

Checking previous logs this seems to stem from a cpsrvd issue (botched update causing out of memory??). The VPS has been running fine and the website is still up, which is kind of why I'm only noticing this now as I haven't had any warnings about server downtime, etc.

The VPS doesn't seem to be under any heavy load or running out of resources:

CPU Load: average 0.10
MEM: 26.3% (3.74G total)
SWAP: 3.0% (1024M total)

Following are logs for each incident:

cpsrvd problem (2013-12-22 00:44:01)
Code: 
[12/22/2013:00:44:01 -0000] info [cpsrvd] reloading config based on -HUP signal
[2013-12-22 00:44:22 +0000] warn [cpsessetup] Failed to create: Cpanel::Cpses::Setup::Postgres: could not determine postgres user at /usr/local/cpanel/Cpanel/Cpses/Setup/Postgres.pm line 21.
 at /usr/local/cpanel/bin/cpsessetup line 99
	bin::cpsessetup::process_cpses_setup_modules() called at /usr/local/cpanel/bin/cpsessetup line 34
	bin::cpsessetup::script('bin::cpsessetup') called at /usr/local/cpanel/bin/cpsessetup line 24
[2013-12-22 00:44:34 +0000] die [realadduser] User cpanel already exists
[2013-12-22 00:44:34 +0000] die [realadduser] User ftp already exists
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
	 "/usr/local/cpanel/base/frontend/x3/branding/the_beach/heading_sprites_bg_snap_to_smallest_width.png".
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
	 "/usr/local/cpanel/base/frontend/x3/branding/motor_city/ui_sprites_bg_snap_to_smallest_width.png".
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
	 "/usr/local/cpanel/base/frontend/x3/branding/crimson_smoke/heading_sprites_bg_snap_to_smallest_width.png".
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
	 "/usr/local/cpanel/base/frontend/x3mail/branding/the_beach/heading_sprites_bg_snap_to_smallest_width.png".
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
	 "/usr/local/cpanel/base/frontend/x3mail/branding/motor_city/ui_sprites_bg_snap_to_smallest_width.png".
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
	 "/usr/local/cpanel/base/frontend/x3mail/branding/crimson_smoke/heading_sprites_bg_snap_to_smallest_width.png".
[2013-12-22 01:00:56 +0000] warn [whostmgr2] Failed to locate postgresql CLI application
Restarting cpsrvd daemon process 2011 via /usr/local/cpanel/cpsrvd-ssl
==> cpsrvd 11.40.1.7 started
==> cpsrvd: preloaded ssl_write_all!
==> cpsrvd: loading security policy....Done
==> cpsrvd: Setting up native SSL support ... Done
==> cpsrvd: transfered port bindings
==> cpsrvd: bound to ports
[B]Out of memory![/B]
From then on (2013-12-22 00:44:01) I initially had the segfault with cpsrvd.so, then numerous page allocation failrues:

cpsrvd.so segfault (2013-12-23 03:52:21)
Code: 
Dec 23 03:52:21 442248 kernel: cpsrvd.so[21715]: segfault at 0 ip 000000361248eff2 sp 00007fff8e325e50 error 4 in libperl.so[3612400000+cf000]
Dec 23 03:52:27 442248 abrtd: Directory 'ccpp-2013-12-23-03:52:21-21715' creation detected
Dec 23 03:52:27 442248 abrt[21724]: Saved core dump of pid 21715 (/usr/local/cpanel/perl/Cpanel/cpsrvd.so) to /var/spool/abrt/ccpp-2013-12-23-03:52:21-21715 (483356672 bytes)
Dec 23 03:52:28 442248 abrtd: Executable '/usr/local/cpanel/perl/Cpanel/cpsrvd.so' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Dec 23 03:52:28 442248 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2013-12-23-03:52:21-21715' exited with 1
Dec 23 03:52:28 442248 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2013-12-23-03:52:21-21715'
page allocation failure: swapper (2013-12-23 12:01:03)
Code: 
Dec 23 12:01:03 442248 kernel: swapper: page allocation failure. order:1, mode:0x20
Dec 23 12:01:03 442248 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 12:01:03 442248 kernel: Call Trace:
Dec 23 12:01:03 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 12:01:03 442248 kernel: [<ffffffffa0115800>] ? virtnet_remove+0x11/0x3f [virtio_net]
Dec 23 12:01:03 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 12:01:03 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 12:01:03 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 12:01:03 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 12:01:03 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 12:01:03 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 12:01:03 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 12:01:03 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 12:01:03 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 12:01:03 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 12:01:03 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 12:01:03 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 12:01:03 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 12:01:03 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 12:01:03 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 12:01:03 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 12:01:03 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 12:01:03 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 12:01:03 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 12:01:03 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 12:01:03 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 12:01:03 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 12:01:03 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 12:01:03 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 12:01:03 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 12:01:03 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 12:01:03 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 12:01:03 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 12:01:03 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 12:01:03 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 12:01:03 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 12:01:03 442248 kernel: <EOI>  [<ffffffff810387db>] ? native_safe_halt+0xb/0x10
Dec 23 12:01:03 442248 kernel: [<ffffffff810149cd>] ? default_idle+0x4d/0xb0
Dec 23 12:01:03 442248 kernel: [<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
Dec 23 12:01:03 442248 kernel: [<ffffffff814e4baa>] ? rest_init+0x7a/0x80
Dec 23 12:01:03 442248 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
Dec 23 12:01:03 442248 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
Dec 23 12:01:03 442248 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
page allocation failure: kswapd0 (2013-12-23 20:01:00)
Code: 
Dec 23 20:01:00 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
Dec 23 20:01:00 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 20:01:00 442248 kernel: Call Trace:
Dec 23 20:01:00 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 20:01:00 442248 kernel: [<ffffffff8143b600>] ? dev_hard_start_xmit+0x1e0/0x3f0
Dec 23 20:01:00 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 20:01:00 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 20:01:00 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 20:01:00 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 20:01:00 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 20:01:00 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 20:01:00 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 20:01:00 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 20:01:00 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 20:01:00 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 20:01:00 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 20:01:00 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 20:01:00 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 20:01:00 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 20:01:00 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 20:01:00 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 20:01:00 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 20:01:00 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 20:01:00 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 20:01:00 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 20:01:00 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 20:01:00 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 20:01:00 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 23 20:01:00 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 20:01:00 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 20:01:00 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 20:01:00 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 20:01:00 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
Dec 23 20:01:00 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 20:01:00 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 20:01:00 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 20:01:00 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 20:01:00 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 20:01:00 442248 kernel: <EOI>  [<ffffffff81500982>] ? _spin_lock+0x12/0x30
Dec 23 20:01:00 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
Dec 23 20:01:00 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
Dec 23 20:01:00 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
Dec 23 20:01:00 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
Dec 23 20:01:00 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
Dec 23 20:01:00 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
Dec 23 20:01:00 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
Dec 23 20:01:00 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
Dec 23 20:01:00 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
Dec 23 20:01:00 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
Dec 23 20:01:00 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
Dec 23 20:01:00 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
Dec 23 20:01:00 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
Dec 23 20:01:00 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
Dec 23 20:01:00 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
Dec 23 20:01:00 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Dec 23 20:01:00 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
Dec 23 20:01:00 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 20:01:00 442248 kernel: Call Trace:
Dec 23 20:01:00 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 20:01:00 442248 kernel: [<ffffffff8102b400>] ? lapic_next_event+0x10/0x30
Dec 23 20:01:00 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 20:01:00 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 20:01:00 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 20:01:00 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 20:01:00 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 20:01:00 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 20:01:00 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 20:01:01 442248 kernel: <EOI>  [<ffffffff81500982>] ? _spin_lock+0x12/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Dec 23 20:01:01 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
Dec 23 20:01:01 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 20:01:01 442248 kernel: Call Trace:
Dec 23 20:01:01 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 20:01:01 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 20:01:01 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 20:01:01 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 20:01:01 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 20:01:01 442248 kernel: <EOI>  [<ffffffff81500982>] ? _spin_lock+0x12/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Dec 23 20:01:01 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
Dec 23 20:01:01 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 20:01:01 442248 kernel: Call Trace:
Dec 23 20:01:01 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 20:01:01 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 20:01:01 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 20:01:01 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 20:01:01 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 20:01:01 442248 kernel: <EOI>  [<ffffffff81500982>] ? _spin_lock+0x12/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Dec 23 20:01:01 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
Dec 23 20:01:01 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 20:01:01 442248 kernel: Call Trace:
Dec 23 20:01:01 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 20:01:01 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 20:01:01 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 20:01:01 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 20:01:01 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 20:01:01 442248 kernel: <EOI>  [<ffffffff81500982>] ? _spin_lock+0x12/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
page allocation failure: swapper (2013-12-26 12:00:31)
Code: 
Dec 26 12:00:31 442248 kernel: swapper: page allocation failure. order:1, mode:0x20
Dec 26 12:00:31 442248 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 26 12:00:31 442248 kernel: Call Trace:
Dec 26 12:00:31 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 26 12:00:31 442248 kernel: [<ffffffff8143b600>] ? dev_hard_start_xmit+0x1e0/0x3f0
Dec 26 12:00:31 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 26 12:00:31 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 26 12:00:31 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 26 12:00:31 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 26 12:00:31 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 26 12:00:31 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 26 12:00:31 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 26 12:00:31 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 26 12:00:31 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 26 12:00:31 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 26 12:00:31 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 26 12:00:31 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 26 12:00:31 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 26 12:00:31 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 26 12:00:31 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 26 12:00:31 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 26 12:00:31 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 26 12:00:31 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 26 12:00:31 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 26 12:00:31 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 26 12:00:31 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 26 12:00:31 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 26 12:00:31 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 26 12:00:31 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 26 12:00:31 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 26 12:00:31 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 26 12:00:31 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 26 12:00:31 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 26 12:00:31 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 26 12:00:31 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 26 12:00:31 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 26 12:00:31 442248 kernel: <EOI>  [<ffffffff810387db>] ? native_safe_halt+0xb/0x10
Dec 26 12:00:31 442248 kernel: [<ffffffff810149cd>] ? default_idle+0x4d/0xb0
Dec 26 12:00:31 442248 kernel: [<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
Dec 26 12:00:31 442248 kernel: [<ffffffff814e4baa>] ? rest_init+0x7a/0x80
Dec 26 12:00:31 442248 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
Dec 26 12:00:31 442248 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
Dec 26 12:00:31 442248 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
Dec 26 12:00:31 442248 kernel: swapper: page allocation failure. order:1, mode:0x20
Dec 26 12:00:31 442248 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 26 12:00:31 442248 kernel: Call Trace:
Dec 26 12:00:31 442248 kernel: <IRQ>  [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 26 12:00:31 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 26 12:00:31 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 26 12:00:31 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 26 12:00:31 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 26 12:00:31 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 26 12:00:31 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 26 12:00:31 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 26 12:00:31 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 26 12:00:31 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 26 12:00:31 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 26 12:00:31 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 26 12:00:31 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 26 12:00:31 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 26 12:00:31 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 26 12:00:31 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 26 12:00:31 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 26 12:00:31 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 26 12:00:31 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 26 12:00:31 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 26 12:00:31 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 26 12:00:31 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 26 12:00:31 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 26 12:00:31 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 26 12:00:31 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 26 12:00:31 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 26 12:00:31 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 26 12:00:31 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 26 12:00:31 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 26 12:00:31 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 26 12:00:31 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 26 12:00:31 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 26 12:00:31 442248 kernel: <EOI>  [<ffffffff810387db>] ? native_safe_halt+0xb/0x10
Dec 26 12:00:31 442248 kernel: [<ffffffff810149cd>] ? default_idle+0x4d/0xb0
Dec 26 12:00:31 442248 kernel: [<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
Dec 26 12:00:31 442248 kernel: [<ffffffff814e4baa>] ? rest_init+0x7a/0x80
Dec 26 12:00:31 442248 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
Dec 26 12:00:31 442248 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
Dec 26 12:00:31 442248 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
page allocation failure: imap (2013-12-28 00:36:05)
Code: 
Dec 28 00:36:05 442248 kernel: imap: page allocation failure. order:1, mode:0x20
Dec 28 00:36:05 442248 kernel: Pid: 6126, comm: imap Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 28 00:36:05 442248 kernel: Call Trace:
Dec 28 00:36:05 442248 kernel: [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 28 00:36:05 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 28 00:36:05 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 28 00:36:05 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 28 00:36:05 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 28 00:36:05 442248 kernel: [<ffffffff8116354a>] ? kmem_cache_alloc_notrace+0xfa/0x130
Dec 28 00:36:05 442248 kernel: [<ffffffff810cddea>] ? audit_log_d_path+0x5a/0x140
Dec 28 00:36:05 442248 kernel: [<ffffffff81237373>] ? common_lsm_audit+0x2e3/0x540
Dec 28 00:36:05 442248 kernel: [<ffffffff8121a4e3>] ? avc_audit+0x73/0xe0
Dec 28 00:36:05 442248 kernel: [<ffffffff8121ac2f>] ? avc_has_perm_noaudit+0x3df/0x470
Dec 28 00:36:05 442248 kernel: [<ffffffff81189b1f>] ? do_lookup+0x9f/0x230
Dec 28 00:36:05 442248 kernel: [<ffffffff8121ad31>] ? avc_has_perm+0x71/0x90
Dec 28 00:36:05 442248 kernel: [<ffffffff8121c734>] ? inode_has_perm+0x54/0xa0
Dec 28 00:36:05 442248 kernel: [<ffffffff8119a960>] ? mntput_no_expire+0x30/0x110
Dec 28 00:36:05 442248 kernel: [<ffffffff81188961>] ? path_put+0x31/0x40
Dec 28 00:36:05 442248 kernel: [<ffffffff8121cd0b>] ? dentry_has_perm+0x5b/0x80
Dec 28 00:36:05 442248 kernel: [<ffffffff8121a550>] ? avc_audit_pre_callback+0x0/0x130
Dec 28 00:36:05 442248 kernel: [<ffffffff8121a680>] ? avc_audit_post_callback+0x0/0x150
Dec 28 00:36:05 442248 kernel: [<ffffffff8121cdbc>] ? selinux_inode_getattr+0x2c/0x30
Dec 28 00:36:05 442248 kernel: [<ffffffff812147d3>] ? security_inode_getattr+0x23/0x30
Dec 28 00:36:05 442248 kernel: [<ffffffff81180a4f>] ? vfs_getattr+0x2f/0x80
Dec 28 00:36:05 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 28 00:36:05 442248 kernel: [<ffffffff81180b00>] ? vfs_fstatat+0x60/0x80
Dec 28 00:36:05 442248 kernel: [<ffffffff81180c4b>] ? vfs_stat+0x1b/0x20
Dec 28 00:36:05 442248 kernel: [<ffffffff81180c74>] ? sys_newstat+0x24/0x50
Dec 28 00:36:05 442248 kernel: [<ffffffff810d6d42>] ? audit_syscall_entry+0x272/0x2a0
Dec 28 00:36:05 442248 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
Dec 28 00:36:05 442248 kernel: Mem-Info:
Dec 28 00:36:05 442248 kernel: Node 0 DMA per-cpu:
Dec 28 00:36:05 442248 kernel: CPU    0: hi:    0, btch:   1 usd:   0
Dec 28 00:36:05 442248 kernel: CPU    1: hi:    0, btch:   1 usd:   0
Dec 28 00:36:05 442248 kernel: Node 0 DMA32 per-cpu:
Dec 28 00:36:05 442248 kernel: CPU    0: hi:  186, btch:  31 usd: 159
Dec 28 00:36:05 442248 kernel: CPU    1: hi:  186, btch:  31 usd: 165
Dec 28 00:36:05 442248 kernel: Node 0 Normal per-cpu:
Dec 28 00:36:05 442248 kernel: CPU    0: hi:  186, btch:  31 usd:  81
Dec 28 00:36:05 442248 kernel: CPU    1: hi:  186, btch:  31 usd: 170
Dec 28 00:36:05 442248 kernel: active_anon:92353 inactive_anon:48818 isolated_anon:0
Dec 28 00:36:05 442248 kernel: active_file:325354 inactive_file:347522 isolated_file:32
Dec 28 00:36:05 442248 kernel: unevictable:0 dirty:470 writeback:0 unstable:0
Dec 28 00:36:05 442248 kernel: free:62119 slab_reclaimable:63385 slab_unreclaimable:21855
Dec 28 00:36:05 442248 kernel: mapped:6554 shmem:59 pagetables:2847 bounce:0
Dec 28 00:36:05 442248 kernel: Node 0 DMA free:15732kB min:248kB low:308kB high:372kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 28 00:36:05 442248 kernel: lowmem_reserve[]: 0 3512 4017 4017
Dec 28 00:36:05 442248 kernel: Node 0 DMA32 free:236080kB min:58868kB low:73584kB high:88300kB active_anon:313744kB inactive_anon:96472kB active_file:1204912kB inactive_file:1277968kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3596500kB mlocked:0kB dirty:1748kB writeback:0kB mapped:22168kB shmem:224kB slab_reclaimable:209972kB slab_unreclaimable:34260kB kernel_stack:616kB pagetables:3384kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 28 00:36:05 442248 kernel: lowmem_reserve[]: 0 0 505 505
Dec 28 00:36:05 442248 kernel: Node 0 Normal free:15760kB min:8464kB low:10580kB high:12696kB active_anon:55668kB inactive_anon:98800kB active_file:95192kB inactive_file:95320kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:517120kB mlocked:0kB dirty:132kB writeback:0kB mapped:4048kB shmem:12kB slab_reclaimable:42764kB slab_unreclaimable:53160kB kernel_stack:888kB pagetables:8004kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 28 00:36:05 442248 kernel: lowmem_reserve[]: 0 0 0 0
Dec 28 00:36:05 442248 kernel: Node 0 DMA: 3*4kB 1*8kB 2*16kB 2*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15732kB
Dec 28 00:36:05 442248 kernel: Node 0 DMA32: 45803*4kB 5940*8kB 832*16kB 41*32kB 10*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 246124kB
Dec 28 00:36:05 442248 kernel: Node 0 Normal: 3219*4kB 784*8kB 13*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 19356kB
Dec 28 00:36:05 442248 kernel: 667331 total pagecache pages
Dec 28 00:36:05 442248 kernel: 2033 pages in swap cache
Dec 28 00:36:05 442248 kernel: Swap cache stats: add 429836, delete 427803, find 68873295/68917309
Dec 28 00:36:05 442248 kernel: Free swap  = 1023284kB
Dec 28 00:36:05 442248 kernel: Total swap = 1048568kB
Dec 28 00:36:05 442248 kernel: 1048575 pages RAM
Dec 28 00:36:05 442248 kernel: 67802 pages reserved
Dec 28 00:36:05 442248 kernel: 272595 pages shared
Dec 28 00:36:05 442248 kernel: 664710 pages non-shared

The last packages to be updated were:
Code: 
Dec 18 00:35:32 442248 yum[3703]: Updated: yum-3.2.29-43.el6.centos.noarch
Dec 20 00:36:19 442248 yum[6213]: Updated: openssl-1.0.1e-16.el6_5.1.x86_64
Dec 20 00:36:20 442248 yum[6213]: Updated: openssl-devel-1.0.1e-16.el6_5.1.x86_64
Dec 20 00:36:20 442248 yum[6213]: Updated: systemtap-runtime-2.3-4.el6_5.x86_64
Dec 21 00:35:35 442248 yum[23604]: Updated: nss-sysinit-3.15.3-3.el6_5.x86_64
Dec 21 00:35:36 442248 yum[23604]: Updated: nss-3.15.3-3.el6_5.x86_64
Dec 21 00:35:36 442248 yum[23604]: Updated: nss-tools-3.15.3-3.el6_5.x86_64
Dec 21 00:35:39 442248 yum[23604]: Updated: ca-certificates-2013.1.95-65.1.el6_5.noarch
Dec 21 00:35:39 442248 yum[23604]: Updated: pixman-0.26.2-5.1.el6_5.x86_64
Dec 21 00:35:42 442248 yum[23604]: Updated: tzdata-2013i-1.el6.noarch
Dec 22 00:41:51 442248 yum[17595]: Updated: iftop-1.0-0.6.pre2.el6.x86_64
I'm not too sure where to begin looking to fix this. Could it be a problem that stems from a failed cpsrvd update? I am also thinking about hardware memory failures, but wanted to check here first.
 
cPanelMichael

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,880
2,270
463
Hello :)

Could you let us know the version of cPanel installed on this system? You can find out the version via:

Code: 
cat /usr/local/cpanel/version
Try running the following command manually:

Code: 
/usr/local/cpanel/perl/Cpanel/cpsrvd.so
Does it complete successfully, or does it hang and continue to run?

Thank you.
 
J

JerryJ.

Member
May 13, 2013
14
0
1
cPanel Access Level
Root Administrator
If your version reports as 11.40.1.7, please run upcp. There was a bug in build 7 which caused cpsrvd.so to segfault.

This is resolved in 11.40.1.8

You can find the change log at http://docs.cpanel.net/twiki/bin/view/AllDocumentation/ChangeLog/CPanelVersion1140

The case number is: 86349

You can also delete any core dumps in /usr/local/cpanel/base.

You can use this command:

rm -f /usr/local/cpanel/base/core.*

IMPORTANT: Do not delete /usr/local/cpanel/base/core, this is not a core dump

:)
 
S

s2s

Member
Nov 21, 2013
20
1
1
cPanel Access Level
Root Administrator
Thanks for getting back guys.

This issue looks to have resolved itself, the last 2 logs don't contain any issues and my current version is : 11.40.1.8
 
You must log in or register to reply here.
Thread starter Similar threads Forum Replies Date
Spirogg SOLVED Prevent cpsrvd from serving standard HTTP ports ? I have this setting ON, could be the reason why AutoSSL is not working? Server Management 2
M Server status HTTPD and cpsrvd status pending Server Management 7
K CPSRVD Killing Memory Server Management 2
A cPanel cannot be executed standalone. It must be run from cpsrvd or xml-api. Server Management 1
L The service “cpsrvd” appears to be down. When CSF enable. Server Management 11
Similar threads
SOLVED Prevent cpsrvd from serving standard HTTP ports ? I have this setting ON, could be the reason why AutoSSL is not working?
Server status HTTPD and cpsrvd status pending
CPSRVD Killing Memory
cPanel cannot be executed standalone. It must be run from cpsrvd or xml-api.
The service “cpsrvd” appears to be down. When CSF enable.
Top Bottom