The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Unusual lockup from attack.

Discussion in 'General Discussion' started by DesigningKnight, Jul 5, 2009.

  1. DesigningKnight

    DesigningKnight Registered

    Joined:
    May 14, 2009
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    First off, my server is running the latest cpanel and the latest updates. Last night, my server completely froze up, unresponsive. I did some checking in the messsages log, and this is what I found. The time this occurred took place exactly at the time the server was unreachable on the network. This repeated over and over until I force rebooted the server from the APC.

    Jul 4 23:56:55 spiritfire kernel: BUG: soft lockup - CPU#1 stuck for 10s! [pop3login:1298]
    Jul 4 23:56:55 spiritfire kernel:
    Jul 4 23:56:55 spiritfire kernel: Pid: 1298, comm: pop3login
    Jul 4 23:56:55 spiritfire kernel: EIP: 0060:[<f8b096da>] CPU: 1
    Jul 4 23:56:55 spiritfire kernel: EIP is at ipt_do_table+0x287/0x2c9 [ip_tables]
    Jul 4 23:56:55 spiritfire kernel: EFLAGS: 00000202 Not tainted (2.6.18-128.1.6.el5 #1)
    Jul 4 23:56:55 spiritfire kernel: EAX: 00000000 EBX: d8f4e8d8 ECX: 00000000 EDX: f8b20001
    Jul 4 23:56:55 spiritfire kernel: ESI: f8b20de4 EDI: d8f4e8d8 EBP: 00000070 DS: 007b ES: 007b
    Jul 4 23:56:55 spiritfire kernel: CR0: 8005003b CR2: 006dd050 CR3: 316bd000 CR4: 000006d0
    Jul 4 23:56:55 spiritfire kernel: [<f8aa4055>] ipt_local_out_hook+0x55/0x5f [iptable_filter]
    Jul 4 23:56:55 spiritfire kernel: [<c05c9a10>] nf_iterate+0x30/0x61
    Jul 4 23:56:55 spiritfire kernel: [<c05d0f80>] dst_output+0x0/0x7
    Jul 4 23:56:55 spiritfire kernel: [<c05c9b36>] nf_hook_slow+0x3a/0x90
    Jul 4 23:56:55 spiritfire kernel: [<c05d0f80>] dst_output+0x0/0x7
    Jul 4 23:56:55 spiritfire kernel: [<c05d327c>] ip_queue_xmit+0x3ba/0x40b
    Jul 4 23:56:55 spiritfire kernel: [<c05d0f80>] dst_output+0x0/0x7
    Jul 4 23:56:55 spiritfire kernel: [<c04744c5>] __find_get_block+0x15c/0x166
    Jul 4 23:56:55 spiritfire kernel: [<c04744ff>] __getblk+0x30/0x27a
    Jul 4 23:56:55 spiritfire kernel: [<c05e0e50>] tcp_transmit_skb+0x5c7/0x5f5
    Jul 4 23:56:55 spiritfire kernel: [<c05e2744>] __tcp_push_pending_frames+0x69c/0x761
    Jul 4 23:56:55 spiritfire kernel: [<c05d8d2e>] tcp_sendmsg+0x8da/0x9e9
    Jul 4 23:56:55 spiritfire kernel: [<c05efe24>] inet_sendmsg+0x35/0x3f
    Jul 4 23:56:55 spiritfire kernel: [<c05ab064>] do_sock_write+0xa3/0xaa
    Jul 4 23:56:55 spiritfire kernel: [<c05ab4ea>] sock_aio_write+0x53/0x61
    Jul 4 23:56:55 spiritfire kernel: [<c04723b6>] do_sync_write+0xb6/0xf1
    Jul 4 23:56:55 spiritfire kernel: [<c043465f>] autoremove_wake_function+0x0/0x2d
    Jul 4 23:56:55 spiritfire kernel: [<c0472c80>] vfs_write+0xb2/0x143
    Jul 4 23:56:55 spiritfire kernel: [<c0473261>] sys_write+0x3c/0x63
    Jul 4 23:56:55 spiritfire kernel: [<c0404f17>] syscall_call+0x7/0xb
    Jul 4 23:56:55 spiritfire kernel: =======================

    I also see that in the logs, preceeding that for about 5 minutes is a huge brute force attack to attempt to gain entry to the server. (this is a portion of it)

    Jul 4 23:56:43 spiritfire cphulkd[1290]: Connection service=system ip= port= user=carol blocked by cphulkd (Too many failures for this username numfailed=6 max=2)
    Jul 4 23:56:44 spiritfire cphulkd[1293]: Connection service=system ip= port= user=changeme blocked by cphulkd (Too many failures for this username numfailed=13 max=2)
    Jul 4 23:56:45 spiritfire cphulkd[1296]: Connection service=system ip= port= user=alice blocked by cphulkd (Too many failures for this username numfailed=3 max=2)

    Noting that the lockup appeared on the pop3login, I checked the mail logs for the time of the attack and found this, which matches with the above log:

    Jul 4 23:56:42 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: LOGIN FAILED, user=woody, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: LOGIN FAILED, user=loretta, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:42 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:43 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:43 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:43 spiritfire pop3d: LOGIN FAILED, user=service, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:43 spiritfire pop3d: LOGIN FAILED, user=master, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:43 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:43 spiritfire pop3d: LOGIN FAILED, user=carol, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:43 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:43 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:44 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:44 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:44 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:44 spiritfire pop3d: LOGIN FAILED, user=changeme, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:44 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:44 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:44 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:45 spiritfire pop3d: LOGIN FAILED, user=alice, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:45 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:45 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:45 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:45 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:45 spiritfire pop3d: LOGOUT, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:45 spiritfire pop3d: Disconnected, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:45 spiritfire pop3d: Connection, ip=[::ffff:65.64.89.245]
    Jul 4 23:56:53 spiritfire pop3d: LOGIN FAILED, user=test , ip=[::ffff:65.64.89.245]
    Jul 4 23:56:53 spiritfire pop3d: authentication error: Input/output error

    (there was actually more, but I had to cut some due to posting limits)

    So what I'm deducing is the lockup of the server came from a brute force attack on the email server. Has anyone else seen anything like this happening?
     
  2. Voltar

    Voltar Well-Known Member

    Joined:
    Apr 30, 2007
    Messages:
    269
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    Bakersfield, California
    You're getting a lot of failed logins in quick succession, so yes someone seems to be brute forcing you.

    Do you have a firewall installed? If so you might want to look into rate limiting the connections, or install CSF if you want a complete solution.
     

Share This Page