Hey guys, so I just came back from Christmas holidays and got to reading through the server logs.
The past few log reports (emailed) contain warnings about page allocation failures:
- kernel: swapper: page allocation failure. order:1, mode:0x20
- kernel: kswapd0: page allocation failure. order:1, mode:0x20
- kernel: imap: page allocation failure. order:1, mode:0x20
Checking previous logs this seems to stem from a cpsrvd issue (botched update causing out of memory??). The VPS has been running fine and the website is still up, which is kind of why I'm only noticing this now as I haven't had any warnings about server downtime, etc.
The VPS doesn't seem to be under any heavy load or running out of resources:
CPU Load: average 0.10
MEM: 26.3% (3.74G total)
SWAP: 3.0% (1024M total)
Following are logs for each incident:
cpsrvd problem (2013-12-22 00:44:01)
From then on (2013-12-22 00:44:01) I initially had the segfault with cpsrvd.so, then numerous page allocation failrues:
cpsrvd.so segfault (2013-12-23 03:52:21)
page allocation failure: swapper (2013-12-23 12:01:03)
page allocation failure: kswapd0 (2013-12-23 20:01:00)
page allocation failure: swapper (2013-12-26 12:00:31)
page allocation failure: imap (2013-12-28 00:36:05)
The last packages to be updated were:
I'm not too sure where to begin looking to fix this. Could it be a problem that stems from a failed cpsrvd update? I am also thinking about hardware memory failures, but wanted to check here first.
The past few log reports (emailed) contain warnings about page allocation failures:
- kernel: swapper: page allocation failure. order:1, mode:0x20
- kernel: kswapd0: page allocation failure. order:1, mode:0x20
- kernel: imap: page allocation failure. order:1, mode:0x20
Checking previous logs this seems to stem from a cpsrvd issue (botched update causing out of memory??). The VPS has been running fine and the website is still up, which is kind of why I'm only noticing this now as I haven't had any warnings about server downtime, etc.
The VPS doesn't seem to be under any heavy load or running out of resources:
CPU Load: average 0.10
MEM: 26.3% (3.74G total)
SWAP: 3.0% (1024M total)
Following are logs for each incident:
cpsrvd problem (2013-12-22 00:44:01)
Code:
[12/22/2013:00:44:01 -0000] info [cpsrvd] reloading config based on -HUP signal
[2013-12-22 00:44:22 +0000] warn [cpsessetup] Failed to create: Cpanel::Cpses::Setup::Postgres: could not determine postgres user at /usr/local/cpanel/Cpanel/Cpses/Setup/Postgres.pm line 21.
at /usr/local/cpanel/bin/cpsessetup line 99
bin::cpsessetup::process_cpses_setup_modules() called at /usr/local/cpanel/bin/cpsessetup line 34
bin::cpsessetup::script('bin::cpsessetup') called at /usr/local/cpanel/bin/cpsessetup line 24
[2013-12-22 00:44:34 +0000] die [realadduser] User cpanel already exists
[2013-12-22 00:44:34 +0000] die [realadduser] User ftp already exists
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
"/usr/local/cpanel/base/frontend/x3/branding/the_beach/heading_sprites_bg_snap_to_smallest_width.png".
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
"/usr/local/cpanel/base/frontend/x3/branding/motor_city/ui_sprites_bg_snap_to_smallest_width.png".
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
"/usr/local/cpanel/base/frontend/x3/branding/crimson_smoke/heading_sprites_bg_snap_to_smallest_width.png".
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
"/usr/local/cpanel/base/frontend/x3mail/branding/the_beach/heading_sprites_bg_snap_to_smallest_width.png".
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
"/usr/local/cpanel/base/frontend/x3mail/branding/motor_city/ui_sprites_bg_snap_to_smallest_width.png".
This is exceeding 8192 pixels and color palette issue may be expected when its associated sprite file is used:
"/usr/local/cpanel/base/frontend/x3mail/branding/crimson_smoke/heading_sprites_bg_snap_to_smallest_width.png".
[2013-12-22 01:00:56 +0000] warn [whostmgr2] Failed to locate postgresql CLI application
Restarting cpsrvd daemon process 2011 via /usr/local/cpanel/cpsrvd-ssl
==> cpsrvd 11.40.1.7 started
==> cpsrvd: preloaded ssl_write_all!
==> cpsrvd: loading security policy....Done
==> cpsrvd: Setting up native SSL support ... Done
==> cpsrvd: transfered port bindings
==> cpsrvd: bound to ports
[B]Out of memory![/B]
cpsrvd.so segfault (2013-12-23 03:52:21)
Code:
Dec 23 03:52:21 442248 kernel: cpsrvd.so[21715]: segfault at 0 ip 000000361248eff2 sp 00007fff8e325e50 error 4 in libperl.so[3612400000+cf000]
Dec 23 03:52:27 442248 abrtd: Directory 'ccpp-2013-12-23-03:52:21-21715' creation detected
Dec 23 03:52:27 442248 abrt[21724]: Saved core dump of pid 21715 (/usr/local/cpanel/perl/Cpanel/cpsrvd.so) to /var/spool/abrt/ccpp-2013-12-23-03:52:21-21715 (483356672 bytes)
Dec 23 03:52:28 442248 abrtd: Executable '/usr/local/cpanel/perl/Cpanel/cpsrvd.so' doesn't belong to any package and ProcessUnpackaged is set to 'no'
Dec 23 03:52:28 442248 abrtd: 'post-create' on '/var/spool/abrt/ccpp-2013-12-23-03:52:21-21715' exited with 1
Dec 23 03:52:28 442248 abrtd: Deleting problem directory '/var/spool/abrt/ccpp-2013-12-23-03:52:21-21715'
Code:
Dec 23 12:01:03 442248 kernel: swapper: page allocation failure. order:1, mode:0x20
Dec 23 12:01:03 442248 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 12:01:03 442248 kernel: Call Trace:
Dec 23 12:01:03 442248 kernel: <IRQ> [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 12:01:03 442248 kernel: [<ffffffffa0115800>] ? virtnet_remove+0x11/0x3f [virtio_net]
Dec 23 12:01:03 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 12:01:03 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 12:01:03 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 12:01:03 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 12:01:03 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 12:01:03 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 12:01:03 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 12:01:03 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 12:01:03 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 12:01:03 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 12:01:03 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 12:01:03 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 12:01:03 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 12:01:03 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 12:01:03 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 12:01:03 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 12:01:03 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 12:01:03 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 12:01:03 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 12:01:03 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 12:01:03 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 12:01:03 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 12:01:03 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 12:01:03 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 12:01:03 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 12:01:03 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 12:01:03 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 12:01:03 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 12:01:03 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 12:01:03 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 12:01:03 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 12:01:03 442248 kernel: <EOI> [<ffffffff810387db>] ? native_safe_halt+0xb/0x10
Dec 23 12:01:03 442248 kernel: [<ffffffff810149cd>] ? default_idle+0x4d/0xb0
Dec 23 12:01:03 442248 kernel: [<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
Dec 23 12:01:03 442248 kernel: [<ffffffff814e4baa>] ? rest_init+0x7a/0x80
Dec 23 12:01:03 442248 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
Dec 23 12:01:03 442248 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
Dec 23 12:01:03 442248 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
Code:
Dec 23 20:01:00 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
Dec 23 20:01:00 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 20:01:00 442248 kernel: Call Trace:
Dec 23 20:01:00 442248 kernel: <IRQ> [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 20:01:00 442248 kernel: [<ffffffff8143b600>] ? dev_hard_start_xmit+0x1e0/0x3f0
Dec 23 20:01:00 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 20:01:00 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 20:01:00 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 20:01:00 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 20:01:00 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 20:01:00 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 20:01:00 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 20:01:00 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 20:01:00 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 20:01:00 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 20:01:00 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 20:01:00 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 20:01:00 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 20:01:00 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 20:01:00 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 20:01:00 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 20:01:00 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 20:01:00 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 20:01:00 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 20:01:00 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 20:01:00 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 20:01:00 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 20:01:00 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 23 20:01:00 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 20:01:00 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 20:01:00 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 20:01:00 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 20:01:00 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
Dec 23 20:01:00 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 20:01:00 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 20:01:00 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 20:01:00 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 20:01:00 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 20:01:00 442248 kernel: <EOI> [<ffffffff81500982>] ? _spin_lock+0x12/0x30
Dec 23 20:01:00 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
Dec 23 20:01:00 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
Dec 23 20:01:00 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
Dec 23 20:01:00 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
Dec 23 20:01:00 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
Dec 23 20:01:00 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
Dec 23 20:01:00 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
Dec 23 20:01:00 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
Dec 23 20:01:00 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
Dec 23 20:01:00 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
Dec 23 20:01:00 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
Dec 23 20:01:00 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
Dec 23 20:01:00 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
Dec 23 20:01:00 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
Dec 23 20:01:00 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
Dec 23 20:01:00 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Dec 23 20:01:00 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
Dec 23 20:01:00 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 20:01:00 442248 kernel: Call Trace:
Dec 23 20:01:00 442248 kernel: <IRQ> [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 20:01:00 442248 kernel: [<ffffffff8102b400>] ? lapic_next_event+0x10/0x30
Dec 23 20:01:00 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 20:01:00 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 20:01:00 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 20:01:00 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 20:01:00 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 20:01:00 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 20:01:00 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 20:01:01 442248 kernel: <EOI> [<ffffffff81500982>] ? _spin_lock+0x12/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Dec 23 20:01:01 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
Dec 23 20:01:01 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 20:01:01 442248 kernel: Call Trace:
Dec 23 20:01:01 442248 kernel: <IRQ> [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 20:01:01 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 20:01:01 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 20:01:01 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 20:01:01 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 20:01:01 442248 kernel: <EOI> [<ffffffff81500982>] ? _spin_lock+0x12/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Dec 23 20:01:01 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
Dec 23 20:01:01 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 20:01:01 442248 kernel: Call Trace:
Dec 23 20:01:01 442248 kernel: <IRQ> [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 20:01:01 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 20:01:01 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 20:01:01 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 20:01:01 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 20:01:01 442248 kernel: <EOI> [<ffffffff81500982>] ? _spin_lock+0x12/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Dec 23 20:01:01 442248 kernel: kswapd0: page allocation failure. order:1, mode:0x20
Dec 23 20:01:01 442248 kernel: Pid: 38, comm: kswapd0 Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 23 20:01:01 442248 kernel: Call Trace:
Dec 23 20:01:01 442248 kernel: <IRQ> [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 23 20:01:01 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 23 20:01:01 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 23 20:01:01 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 23 20:01:01 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 23 20:01:01 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 23 20:01:01 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 23 20:01:01 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 23 20:01:01 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 23 20:01:01 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 23 20:01:01 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 23 20:01:01 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 23 20:01:01 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 23 20:01:01 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 23 20:01:01 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 23 20:01:01 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 23 20:01:01 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 23 20:01:01 442248 kernel: [<ffffffff81073fbf>] ? __do_softirq+0x11f/0x1e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 23 20:01:01 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 23 20:01:01 442248 kernel: <EOI> [<ffffffff81500982>] ? _spin_lock+0x12/0x30
Dec 23 20:01:01 442248 kernel: [<ffffffffa007a7f8>] ? jbd2_journal_release_jbd_inode+0xc8/0x100 [jbd2]
Dec 23 20:01:01 442248 kernel: [<ffffffffa00a9e6c>] ? ext4_clear_inode+0x3c/0x50 [ext4]
Dec 23 20:01:01 442248 kernel: [<ffffffff8119610c>] ? clear_inode+0xac/0x140
Dec 23 20:01:01 442248 kernel: [<ffffffff811961e0>] ? dispose_list+0x40/0x120
Dec 23 20:01:01 442248 kernel: [<ffffffff81196534>] ? shrink_icache_memory+0x274/0x2e0
Dec 23 20:01:01 442248 kernel: [<ffffffff8112d3ea>] ? shrink_slab+0x12a/0x1a0
Dec 23 20:01:01 442248 kernel: [<ffffffff811302bd>] ? balance_pgdat+0x57d/0x7e0
Dec 23 20:01:01 442248 kernel: [<ffffffff811308d0>] ? isolate_pages_global+0x0/0x350
Dec 23 20:01:01 442248 kernel: [<ffffffff81124a5d>] ? zone_watermark_ok_safe+0xad/0xc0
Dec 23 20:01:01 442248 kernel: [<ffffffff81130656>] ? kswapd+0x136/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81092170>] ? autoremove_wake_function+0x0/0x40
Dec 23 20:01:01 442248 kernel: [<ffffffff81130520>] ? kswapd+0x0/0x3b0
Dec 23 20:01:01 442248 kernel: [<ffffffff81091e06>] ? kthread+0x96/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
Dec 23 20:01:01 442248 kernel: [<ffffffff81091d70>] ? kthread+0x0/0xa0
Dec 23 20:01:01 442248 kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20
Code:
Dec 26 12:00:31 442248 kernel: swapper: page allocation failure. order:1, mode:0x20
Dec 26 12:00:31 442248 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 26 12:00:31 442248 kernel: Call Trace:
Dec 26 12:00:31 442248 kernel: <IRQ> [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 26 12:00:31 442248 kernel: [<ffffffff8143b600>] ? dev_hard_start_xmit+0x1e0/0x3f0
Dec 26 12:00:31 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 26 12:00:31 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 26 12:00:31 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 26 12:00:31 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 26 12:00:31 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 26 12:00:31 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 26 12:00:31 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 26 12:00:31 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 26 12:00:31 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 26 12:00:31 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 26 12:00:31 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 26 12:00:31 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 26 12:00:31 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 26 12:00:31 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 26 12:00:31 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 26 12:00:31 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 26 12:00:31 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 26 12:00:31 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 26 12:00:31 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 26 12:00:31 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 26 12:00:31 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 26 12:00:31 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 26 12:00:31 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 26 12:00:31 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 26 12:00:31 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 26 12:00:31 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 26 12:00:31 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 26 12:00:31 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 26 12:00:31 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 26 12:00:31 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 26 12:00:31 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 26 12:00:31 442248 kernel: <EOI> [<ffffffff810387db>] ? native_safe_halt+0xb/0x10
Dec 26 12:00:31 442248 kernel: [<ffffffff810149cd>] ? default_idle+0x4d/0xb0
Dec 26 12:00:31 442248 kernel: [<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
Dec 26 12:00:31 442248 kernel: [<ffffffff814e4baa>] ? rest_init+0x7a/0x80
Dec 26 12:00:31 442248 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
Dec 26 12:00:31 442248 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
Dec 26 12:00:31 442248 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
Dec 26 12:00:31 442248 kernel: swapper: page allocation failure. order:1, mode:0x20
Dec 26 12:00:31 442248 kernel: Pid: 0, comm: swapper Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 26 12:00:31 442248 kernel: Call Trace:
Dec 26 12:00:31 442248 kernel: <IRQ> [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 26 12:00:31 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 26 12:00:31 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 26 12:00:31 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 26 12:00:31 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 26 12:00:31 442248 kernel: [<ffffffff81163aeb>] ? kmem_cache_alloc+0x11b/0x190
Dec 26 12:00:31 442248 kernel: [<ffffffff8142e278>] ? sk_prot_alloc+0x48/0x1c0
Dec 26 12:00:31 442248 kernel: [<ffffffff8142e542>] ? sk_clone+0x22/0x2e0
Dec 26 12:00:31 442248 kernel: [<ffffffff8147c276>] ? inet_csk_clone+0x16/0xd0
Dec 26 12:00:31 442248 kernel: [<ffffffff81495253>] ? tcp_create_openreq_child+0x23/0x450
Dec 26 12:00:31 442248 kernel: [<ffffffff81492abd>] ? tcp_v4_syn_recv_sock+0x4d/0x310
Dec 26 12:00:31 442248 kernel: [<ffffffff81494ff6>] ? tcp_check_req+0x226/0x460
Dec 26 12:00:31 442248 kernel: [<ffffffff814924db>] ? tcp_v4_do_rcv+0x35b/0x430
Dec 26 12:00:31 442248 kernel: [<ffffffffa0107557>] ? ipv4_confirm+0x87/0x1d0 [nf_conntrack_ipv4]
Dec 26 12:00:31 442248 kernel: [<ffffffff81493d1e>] ? tcp_v4_rcv+0x4fe/0x8d0
Dec 26 12:00:31 442248 kernel: [<ffffffff814718d0>] ? ip_local_deliver_finish+0x0/0x2d0
Dec 26 12:00:31 442248 kernel: [<ffffffff814719ad>] ? ip_local_deliver_finish+0xdd/0x2d0
Dec 26 12:00:31 442248 kernel: [<ffffffff81471c38>] ? ip_local_deliver+0x98/0xa0
Dec 26 12:00:31 442248 kernel: [<ffffffff814710fd>] ? ip_rcv_finish+0x12d/0x440
Dec 26 12:00:31 442248 kernel: [<ffffffff81471685>] ? ip_rcv+0x275/0x350
Dec 26 12:00:31 442248 kernel: [<ffffffff8143adcb>] ? __netif_receive_skb+0x49b/0x6f0
Dec 26 12:00:31 442248 kernel: [<ffffffff8143d048>] ? netif_receive_skb+0x58/0x60
Dec 26 12:00:31 442248 kernel: [<ffffffffa011455d>] ? virtnet_poll+0x5ed/0x8e0 [virtio_net]
Dec 26 12:00:31 442248 kernel: [<ffffffff8143f7a3>] ? net_rx_action+0x103/0x2f0
Dec 26 12:00:31 442248 kernel: [<ffffffffa01130b9>] ? skb_recv_done+0x39/0x40 [virtio_net]
Dec 26 12:00:31 442248 kernel: [<ffffffff81073f61>] ? __do_softirq+0xc1/0x1e0
Dec 26 12:00:31 442248 kernel: [<ffffffff810dbb60>] ? handle_IRQ_event+0x60/0x170
Dec 26 12:00:31 442248 kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 26 12:00:31 442248 kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 26 12:00:31 442248 kernel: [<ffffffff81073d45>] ? irq_exit+0x85/0x90
Dec 26 12:00:31 442248 kernel: [<ffffffff81506365>] ? do_IRQ+0x75/0xf0
Dec 26 12:00:31 442248 kernel: [<ffffffff8100ba53>] ? ret_from_intr+0x0/0x11
Dec 26 12:00:31 442248 kernel: <EOI> [<ffffffff810387db>] ? native_safe_halt+0xb/0x10
Dec 26 12:00:31 442248 kernel: [<ffffffff810149cd>] ? default_idle+0x4d/0xb0
Dec 26 12:00:31 442248 kernel: [<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
Dec 26 12:00:31 442248 kernel: [<ffffffff814e4baa>] ? rest_init+0x7a/0x80
Dec 26 12:00:31 442248 kernel: [<ffffffff81c21f7b>] ? start_kernel+0x424/0x430
Dec 26 12:00:31 442248 kernel: [<ffffffff81c2133a>] ? x86_64_start_reservations+0x125/0x129
Dec 26 12:00:31 442248 kernel: [<ffffffff81c21438>] ? x86_64_start_kernel+0xfa/0x109
Code:
Dec 28 00:36:05 442248 kernel: imap: page allocation failure. order:1, mode:0x20
Dec 28 00:36:05 442248 kernel: Pid: 6126, comm: imap Not tainted 2.6.32-279.14.1.el6.x86_64 #1
Dec 28 00:36:05 442248 kernel: Call Trace:
Dec 28 00:36:05 442248 kernel: [<ffffffff811278ff>] ? __alloc_pages_nodemask+0x77f/0x940
Dec 28 00:36:05 442248 kernel: [<ffffffff81162372>] ? kmem_getpages+0x62/0x170
Dec 28 00:36:05 442248 kernel: [<ffffffff81162f8a>] ? fallback_alloc+0x1ba/0x270
Dec 28 00:36:05 442248 kernel: [<ffffffff811629df>] ? cache_grow+0x2cf/0x320
Dec 28 00:36:05 442248 kernel: [<ffffffff81162d09>] ? ____cache_alloc_node+0x99/0x160
Dec 28 00:36:05 442248 kernel: [<ffffffff8116354a>] ? kmem_cache_alloc_notrace+0xfa/0x130
Dec 28 00:36:05 442248 kernel: [<ffffffff810cddea>] ? audit_log_d_path+0x5a/0x140
Dec 28 00:36:05 442248 kernel: [<ffffffff81237373>] ? common_lsm_audit+0x2e3/0x540
Dec 28 00:36:05 442248 kernel: [<ffffffff8121a4e3>] ? avc_audit+0x73/0xe0
Dec 28 00:36:05 442248 kernel: [<ffffffff8121ac2f>] ? avc_has_perm_noaudit+0x3df/0x470
Dec 28 00:36:05 442248 kernel: [<ffffffff81189b1f>] ? do_lookup+0x9f/0x230
Dec 28 00:36:05 442248 kernel: [<ffffffff8121ad31>] ? avc_has_perm+0x71/0x90
Dec 28 00:36:05 442248 kernel: [<ffffffff8121c734>] ? inode_has_perm+0x54/0xa0
Dec 28 00:36:05 442248 kernel: [<ffffffff8119a960>] ? mntput_no_expire+0x30/0x110
Dec 28 00:36:05 442248 kernel: [<ffffffff81188961>] ? path_put+0x31/0x40
Dec 28 00:36:05 442248 kernel: [<ffffffff8121cd0b>] ? dentry_has_perm+0x5b/0x80
Dec 28 00:36:05 442248 kernel: [<ffffffff8121a550>] ? avc_audit_pre_callback+0x0/0x130
Dec 28 00:36:05 442248 kernel: [<ffffffff8121a680>] ? avc_audit_post_callback+0x0/0x150
Dec 28 00:36:05 442248 kernel: [<ffffffff8121cdbc>] ? selinux_inode_getattr+0x2c/0x30
Dec 28 00:36:05 442248 kernel: [<ffffffff812147d3>] ? security_inode_getattr+0x23/0x30
Dec 28 00:36:05 442248 kernel: [<ffffffff81180a4f>] ? vfs_getattr+0x2f/0x80
Dec 28 00:36:05 442248 kernel: [<ffffffff81039688>] ? pvclock_clocksource_read+0x58/0xd0
Dec 28 00:36:05 442248 kernel: [<ffffffff81180b00>] ? vfs_fstatat+0x60/0x80
Dec 28 00:36:05 442248 kernel: [<ffffffff81180c4b>] ? vfs_stat+0x1b/0x20
Dec 28 00:36:05 442248 kernel: [<ffffffff81180c74>] ? sys_newstat+0x24/0x50
Dec 28 00:36:05 442248 kernel: [<ffffffff810d6d42>] ? audit_syscall_entry+0x272/0x2a0
Dec 28 00:36:05 442248 kernel: [<ffffffff8100b0f2>] ? system_call_fastpath+0x16/0x1b
Dec 28 00:36:05 442248 kernel: Mem-Info:
Dec 28 00:36:05 442248 kernel: Node 0 DMA per-cpu:
Dec 28 00:36:05 442248 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Dec 28 00:36:05 442248 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Dec 28 00:36:05 442248 kernel: Node 0 DMA32 per-cpu:
Dec 28 00:36:05 442248 kernel: CPU 0: hi: 186, btch: 31 usd: 159
Dec 28 00:36:05 442248 kernel: CPU 1: hi: 186, btch: 31 usd: 165
Dec 28 00:36:05 442248 kernel: Node 0 Normal per-cpu:
Dec 28 00:36:05 442248 kernel: CPU 0: hi: 186, btch: 31 usd: 81
Dec 28 00:36:05 442248 kernel: CPU 1: hi: 186, btch: 31 usd: 170
Dec 28 00:36:05 442248 kernel: active_anon:92353 inactive_anon:48818 isolated_anon:0
Dec 28 00:36:05 442248 kernel: active_file:325354 inactive_file:347522 isolated_file:32
Dec 28 00:36:05 442248 kernel: unevictable:0 dirty:470 writeback:0 unstable:0
Dec 28 00:36:05 442248 kernel: free:62119 slab_reclaimable:63385 slab_unreclaimable:21855
Dec 28 00:36:05 442248 kernel: mapped:6554 shmem:59 pagetables:2847 bounce:0
Dec 28 00:36:05 442248 kernel: Node 0 DMA free:15732kB min:248kB low:308kB high:372kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15336kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Dec 28 00:36:05 442248 kernel: lowmem_reserve[]: 0 3512 4017 4017
Dec 28 00:36:05 442248 kernel: Node 0 DMA32 free:236080kB min:58868kB low:73584kB high:88300kB active_anon:313744kB inactive_anon:96472kB active_file:1204912kB inactive_file:1277968kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3596500kB mlocked:0kB dirty:1748kB writeback:0kB mapped:22168kB shmem:224kB slab_reclaimable:209972kB slab_unreclaimable:34260kB kernel_stack:616kB pagetables:3384kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 28 00:36:05 442248 kernel: lowmem_reserve[]: 0 0 505 505
Dec 28 00:36:05 442248 kernel: Node 0 Normal free:15760kB min:8464kB low:10580kB high:12696kB active_anon:55668kB inactive_anon:98800kB active_file:95192kB inactive_file:95320kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:517120kB mlocked:0kB dirty:132kB writeback:0kB mapped:4048kB shmem:12kB slab_reclaimable:42764kB slab_unreclaimable:53160kB kernel_stack:888kB pagetables:8004kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
Dec 28 00:36:05 442248 kernel: lowmem_reserve[]: 0 0 0 0
Dec 28 00:36:05 442248 kernel: Node 0 DMA: 3*4kB 1*8kB 2*16kB 2*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15732kB
Dec 28 00:36:05 442248 kernel: Node 0 DMA32: 45803*4kB 5940*8kB 832*16kB 41*32kB 10*64kB 1*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 246124kB
Dec 28 00:36:05 442248 kernel: Node 0 Normal: 3219*4kB 784*8kB 13*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 19356kB
Dec 28 00:36:05 442248 kernel: 667331 total pagecache pages
Dec 28 00:36:05 442248 kernel: 2033 pages in swap cache
Dec 28 00:36:05 442248 kernel: Swap cache stats: add 429836, delete 427803, find 68873295/68917309
Dec 28 00:36:05 442248 kernel: Free swap = 1023284kB
Dec 28 00:36:05 442248 kernel: Total swap = 1048568kB
Dec 28 00:36:05 442248 kernel: 1048575 pages RAM
Dec 28 00:36:05 442248 kernel: 67802 pages reserved
Dec 28 00:36:05 442248 kernel: 272595 pages shared
Dec 28 00:36:05 442248 kernel: 664710 pages non-shared
The last packages to be updated were:
Code:
Dec 18 00:35:32 442248 yum[3703]: Updated: yum-3.2.29-43.el6.centos.noarch
Dec 20 00:36:19 442248 yum[6213]: Updated: openssl-1.0.1e-16.el6_5.1.x86_64
Dec 20 00:36:20 442248 yum[6213]: Updated: openssl-devel-1.0.1e-16.el6_5.1.x86_64
Dec 20 00:36:20 442248 yum[6213]: Updated: systemtap-runtime-2.3-4.el6_5.x86_64
Dec 21 00:35:35 442248 yum[23604]: Updated: nss-sysinit-3.15.3-3.el6_5.x86_64
Dec 21 00:35:36 442248 yum[23604]: Updated: nss-3.15.3-3.el6_5.x86_64
Dec 21 00:35:36 442248 yum[23604]: Updated: nss-tools-3.15.3-3.el6_5.x86_64
Dec 21 00:35:39 442248 yum[23604]: Updated: ca-certificates-2013.1.95-65.1.el6_5.noarch
Dec 21 00:35:39 442248 yum[23604]: Updated: pixman-0.26.2-5.1.el6_5.x86_64
Dec 21 00:35:42 442248 yum[23604]: Updated: tzdata-2013i-1.el6.noarch
Dec 22 00:41:51 442248 yum[17595]: Updated: iftop-1.0-0.6.pre2.el6.x86_64