Unusual lockup from attack.

[DesigningKnight]

First off, my server is running the latest cpanel and the latest updates. Last night, my server completely froze up, unresponsive. I did some checking in the messsages log, and this is what I found. The time this occurred took place exactly at the time the server was unreachable on the network. This repeated over and over until I force rebooted the server from the APC.

Jul 4 23:56:55 spiritfire kernel: BUG: soft lockup - CPU#1 stuck for 10s! [pop3login:1298]
Jul 4 23:56:55 spiritfire kernel:
Jul 4 23:56:55 spiritfire kernel: Pid: 1298, comm: pop3login
Jul 4 23:56:55 spiritfire kernel: EIP: 0060:[<f8b096da>] CPU: 1
Jul 4 23:56:55 spiritfire kernel: EIP is at ipt_do_table+0x287/0x2c9 [ip_tables]
Jul 4 23:56:55 spiritfire kernel: EFLAGS: 00000202 Not tainted (2.6.18-128.1.6.el5 #1)
Jul 4 23:56:55 spiritfire kernel: EAX: 00000000 EBX: d8f4e8d8 ECX: 00000000 EDX: f8b20001
Jul 4 23:56:55 spiritfire kernel: ESI: f8b20de4 EDI: d8f4e8d8 EBP: 00000070 DS: 007b ES: 007b
Jul 4 23:56:55 spiritfire kernel: CR0: 8005003b CR2: 006dd050 CR3: 316bd000 CR4: 000006d0
Jul 4 23:56:55 spiritfire kernel: [<f8aa4055>] ipt_local_out_hook+0x55/0x5f [iptable_filter]
Jul 4 23:56:55 spiritfire kernel: [<c05c9a10>] nf_iterate+0x30/0x61
Jul 4 23:56:55 spiritfire kernel: [<c05d0f80>] dst_output+0x0/0x7
Jul 4 23:56:55 spiritfire kernel: [<c05c9b36>] nf_hook_slow+0x3a/0x90
Jul 4 23:56:55 spiritfire kernel: [<c05d0f80>] dst_output+0x0/0x7
Jul 4 23:56:55 spiritfire kernel: [<c05d327c>] ip_queue_xmit+0x3ba/0x40b
Jul 4 23:56:55 spiritfire kernel: [<c05d0f80>] dst_output+0x0/0x7
Jul 4 23:56:55 spiritfire kernel: [<c04744c5>] __find_get_block+0x15c/0x166
Jul 4 23:56:55 spiritfire kernel: [<c04744ff>] __getblk+0x30/0x27a
Jul 4 23:56:55 spiritfire kernel: [<c05e0e50>] tcp_transmit_skb+0x5c7/0x5f5
Jul 4 23:56:55 spiritfire kernel: [<c05e2744>] __tcp_push_pending_frames+0x69c/0x761
Jul 4 23:56:55 spiritfire kernel: [<c05d8d2e>] tcp_sendmsg+0x8da/0x9e9
Jul 4 23:56:55 spiritfire kernel: [<c05efe24>] inet_sendmsg+0x35/0x3f
Jul 4 23:56:55 spiritfire kernel: [<c05ab064>] do_sock_write+0xa3/0xaa
Jul 4 23:56:55 spiritfire kernel: [<c05ab4ea>] sock_aio_write+0x53/0x61
Jul 4 23:56:55 spiritfire kernel: [<c04723b6>] do_sync_write+0xb6/0xf1
Jul 4 23:56:55 spiritfire kernel: [<c043465f>] autoremove_wake_function+0x0/0x2d
Jul 4 23:56:55 spiritfire kernel: [<c0472c80>] vfs_write+0xb2/0x143
Jul 4 23:56:55 spiritfire kernel: [<c0473261>] sys_write+0x3c/0x63
Jul 4 23:56:55 spiritfire kernel: [<c0404f17>] syscall_call+0x7/0xb
Jul 4 23:56:55 spiritfire kernel: =======================

I also see that in the logs, preceeding that for about 5 minutes is a huge brute force attack to attempt to gain entry to the server. (this is a portion of it)

Jul 4 23:56:43 spiritfire cphulkd[1290]: Connection service=system ip= port= user=carol blocked by cphulkd (Too many failures for this username numfailed=6 max=2)
Jul 4 23:56:44 spiritfire cphulkd[1293]: Connection service=system ip= port= user=changeme blocked by cphulkd (Too many failures for this username numfailed=13 max=2)
Jul 4 23:56:45 spiritfire cphulkd[1296]: Connection service=system ip= port= user=alice blocked by cphulkd (Too many failures for this username numfailed=3 max=2)

Noting that the lockup appeared on the pop3login, I checked the mail logs for the time of the attack and found this, which matches with the above log:

Jul 4 23:56:42 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: LOGIN FAILED, user=woody, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: LOGIN FAILED, user=loretta, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
Jul 4 23:56:42 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
Jul 4 23:56:43 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
Jul 4 23:56:43 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
Jul 4 23:56:43 spiritfire pop3d: LOGIN FAILED, user=service, ip=[::ffff:65.64.89.245]
Jul 4 23:56:43 spiritfire pop3d: LOGIN FAILED, user=master, ip=[::ffff:65.64.89.245]
Jul 4 23:56:43 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
Jul 4 23:56:43 spiritfire pop3d: LOGIN FAILED, user=carol, ip=[::ffff:65.64.89.245]
Jul 4 23:56:43 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
Jul 4 23:56:43 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
Jul 4 23:56:44 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
Jul 4 23:56:44 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
Jul 4 23:56:44 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
Jul 4 23:56:44 spiritfire pop3d: LOGIN FAILED, user=changeme, ip=[::ffff:65.64.89.245]
Jul 4 23:56:44 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
Jul 4 23:56:44 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
Jul 4 23:56:44 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
Jul 4 23:56:45 spiritfire pop3d: LOGIN FAILED, user=alice, ip=[::ffff:65.64.89.245]
Jul 4 23:56:45 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
Jul 4 23:56:45 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
Jul 4 23:56:45 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
Jul 4 23:56:45 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
Jul 4 23:56:45 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
Jul 4 23:56:45 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
Jul 4 23:56:45 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
Jul 4 23:56:53 spiritfire pop3d: LOGIN FAILED, user=test , ip=[::ffff:65.64.89.245]
Jul 4 23:56:53 spiritfire pop3d: authentication error: Input/output error

(there was actually more, but I had to cut some due to posting limits)

So what I'm deducing is the lockup of the server came from a brute force attack on the email server. Has anyone else seen anything like this happening?

 

[Voltar]

You're getting a lot of failed logins in quick succession, so yes someone seems to be brute forcing you.

Do you have a firewall installed? If so you might want to look into rate limiting the connections, or install CSF if you want a complete solution.