The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Question about lsof output (is this an attack?)

Discussion in 'General Discussion' started by noimad1, Oct 7, 2007.

  1. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16
    Ok,

    First, when I run top I have few processes that look like this:


    2 /usr/local/apache/bin/httpd -DSSL 0 16016 11M 2524 R 25.1 0.5 1653m


    And they have been running for like a day or two.

    Now, when I do a lsof -p 988 it outputs something like this:

    httpd 988 nobody cwd DIR 3,5 8192 7602580 /home/auser/public_html
    httpd 988 nobody rtd DIR 3,6 4096 2 /
    httpd 988 nobody txt REG 3,2 660076 999882 /usr/local/apache/bin/httpd
    httpd 988 nobody mem REG 3,2 241464 639911 /usr/lib/libexpat.so.0.4.0
    httpd 988 nobody mem REG 3,2 72552 1065548 /usr/kerberos/lib/libk5crypto.so.3.0
    httpd 988 nobody mem REG 3,2 7741 885451 /usr/local/apache/libexec/mod_bwlimited.so
    httpd 988 nobody mem REG 3,2 8026 885472 /usr/local/apache/libexec/mod_log_bytes.so
    httpd 988 nobody mem REG 3,2 108363 885190 /usr/local/apache/libexec/mod_security.so
    httpd 988 nobody mem REG 3,6 63292 78981 /lib/libz.so.1.2.2.1
    httpd 988 nobody mem REG 3,2 5540 1065528 /usr/kerberos/lib/libcom_err.so.3.0
    httpd 988 nobody mem REG 3,6 211908 79101 /lib/libssl.so.0.9.7a
    httpd 988 nobody mem REG 3,6 1516255 18049 /lib/tls/libc-2.3.2.so
    httpd 988 nobody mem REG 3,6 976316 78017 /lib/libcrypto.so.0.9.7a
    httpd 988 nobody mem REG 3,6 102480 74025 /lib/ld-2.3.2.so
    httpd 988 nobody mem REG 3,6 13601 77419 /lib/libdl-2.3.2.so
    httpd 988 nobody mem REG 3,6 50783 77428 /lib/libnss_files-2.3.2.so
    httpd 988 nobody mem REG 3,2 22504 639027 /usr/lib/libgdbm.so.2.0.0
    httpd 988 nobody mem REG 3,2 59459 885273 /usr/local/apache/libexec/mod_rewrite.so
    httpd 988 nobody mem REG 3,2 76712 1065539 /usr/kerberos/lib/libgssapi_krb5.so.2.2
    httpd 988 nobody mem REG 3,6 185942 18811 /lib/tls/libm-2.3.2.so
    httpd 988 nobody mem REG 3,2 385252 1066424 /usr/kerberos/lib/libkrb5.so.3.1
    httpd 988 nobody mem REG 3,6 22242 73131 /lib/libcrypt-2.3.2.so
    httpd 988 nobody mem REG 3,2 7857 885473 /usr/local/apache/libexec/mod_auth_passthrough.so
    httpd 988 nobody mem REG 3,2 9533 885462 /usr/local/apache/libexec/mod_expires.so
    httpd 988 nobody mem REG 3,6 75050 73165 /lib/libresolv-2.3.2.so
    httpd 988 nobody mem DEL 0,4 0 /SYSV00000000
    httpd 988 nobody 0r CHR 1,3 89324 /dev/null
    httpd 988 nobody 1w CHR 1,3 89324 /dev/null
    httpd 988 nobody 2w REG 3,2 227031956 296545 /usr/local/apache/logs/error_log
    httpd 988 nobody 3u sock 0,0 1937188 can't identify protocol
    httpd 988 nobody 4w FIFO 0,5 14938 pipe
    httpd 988 nobody 5r FIFO 0,5 14939 pipe
    httpd 988 nobody 6r FIFO 0,5 1518950 pipe
    httpd 988 nobody 7r FIFO 0,5 14940 pipe
    httpd 988 nobody 15w REG 3,2 155502064 295093 /usr/local/apache/logs/audit_log
    httpd 988 nobody 16w REG 3,2 227031956 296545 /usr/local/apache/logs/error_log
    httpd 988 nobody 17u IPv4 13462 TCP *:https (LISTEN)
    httpd 988 nobody 18u IPv4 13463 TCP *:http (LISTEN)
    httpd 988 nobody 19w REG 3,2 0 1101806 /usr/local/apache/domlogs/adomain.com
    httpd 988 nobody 20w REG 3,2 0 1102468 /usr/local/apache/domlogs/adomain.com
    httpd 988 nobody 21w REG 3,2 72 1101526 /usr/local/apache/domlogs/adomain.com
    httpd 988 nobody 22w REG 3,2 6604 1101877 /usr/local/apache/domlogs/adomain.com
    httpd 988 nobody 23w REG 3,2 0 1101807 /usr/local/apache/domlogs/adomain.com
    httpd 988 nobody 24w REG 3,2 0 1101731 /usr/local/apache/domlogs/adomain.com
    httpd 988 nobody 25w REG 3,2 0 1101594 /usr/local/apache/domlogs/adomain.com



    And it continues on with a whole list of files in the domlogs, but I didn't paste them all.

    Now, i'm a little new to the lsof command, so is this something normal to see as an output from that log?

    I'm just trying to figure out what these processes are. They seem to be killing my system as far as load goes.

    I tried to do a strace -p 988, but it just sits there.

    When I go to the WHM and do a cpu stat thing I don't see this process listed at all?

    What else can I do to find out maybe what is running this process? Is there any other way to trace it, or maybe see if it is coming from an IP address or something like that maybe?
     
  2. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16
    The other thing is I can't kill these processes either?
     
  3. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16
    Well I had to reboot to get the server to come back online.

    I was able to kill all http prcoesses, but not the ones listed like above. Then I couldn't restart apache because it would give me this error:

    [Sun Oct 7 17:18:26 2007] [crit] (98)Address already in use: make_sock: could not bind to port 443



    So I'm assuing that they were related. This has happened twice in the past few days. So if anyone has any incite in case it happens again that would be great.
     
  4. mtindor

    mtindor Well-Known Member

    Joined:
    Sep 14, 2004
    Messages:
    1,279
    Likes Received:
    36
    Trophy Points:
    48
    Location:
    inside a catfish
    cPanel Access Level:
    Root Administrator
    Exactly what part of that log are _you_ curious or suspicious about. It sure helps to point out what is triggering _your_ suspicions. So please, is there some particular reason this looks suspicious to you?

    When you run top, those processes you see are the apache listeners (there are a minimum number that will always be in a top listing even if no traffic is hitting your website).

    I don't see anything suspicious. I'll take another look after I make this post. But all looks normal at first glance to me.

    Mike
     
  5. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16
    Mike,

    Thanks for the response. Well my number one issue I have with these processes is the amount of CPU usage they are taking up. There are usually at least 3 of them running, and each is using up anywhere from 25% to 40% of the cpu usage. So my load is upwards of 15.00-20.00. Then after I reboot the server, it goes down to less then .90- 1.00.

    I'm seeing the first part that I posted from the lsof -9 PID

    Then after I do like a "killall -9 httpd" it kills all of my normal httpd processes.

    However, when I try to do a httpd -startssl it says its started, but isn't. When I look in my /usr/local/apache/logs I see that error message about http already being bound to a port. If I try to do a httpd stop it says it isn't running.

    If I try more killall -9 httpd or even killall -9 /user/local/apache/httpd it won't kill those three or four running processes.

    So the only way I can find to get httpd back up and running is to completely reboot the entire server.

    So like I said I'm kinda noob when it comes to some of this stuff, so if there is another log I should look in, please let me know. Or if there is something that seems obvious that I'm missing that would be great as well.

    This has happened at least twice in the past couple of days, to the point where the load practically brings the server down. And I've had a few other times when the server has crashed this week, and I suspect this is the same issue, but where I didn't catch it before the load went out of control and I couldn't ssh in at all....
     
  6. mctDarren

    mctDarren Well-Known Member

    Joined:
    Jan 6, 2004
    Messages:
    664
    Likes Received:
    2
    Trophy Points:
    18
    Location:
    New Jersey
    cPanel Access Level:
    Root Administrator
    Do a netstat -tup and see if you can figure out what has bound itself to port 443.
     
  7. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16

    Ah, that was probably the command i was missing. Ok, i had to reboot the server last time it happened. So I'll watch it again and see what netstat -tup tells me if it happens again.
     
  8. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,449
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Are you run running mod_security and if yes what rulesets? (I'm probably off base here but asking anyway) ;)
     
  9. mctDarren

    mctDarren Well-Known Member

    Joined:
    Jan 6, 2004
    Messages:
    664
    Likes Received:
    2
    Trophy Points:
    18
    Location:
    New Jersey
    cPanel Access Level:
    Root Administrator
    watch "netstat -tup" will show the output of that command every 2 secs if you want to monitor it. You should see it right away if something is bound to it.
     
  10. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16
    Yes I am running mod security. And I pretty much using a ruleset written by by Kris S. - HostMerit.com along with my own custom rules in there as well.
     
  11. Infopro

    Infopro cPanel Sr. Product Evangelist
    Staff Member

    Joined:
    May 20, 2003
    Messages:
    14,449
    Likes Received:
    195
    Trophy Points:
    63
    Location:
    Pennsylvania
    cPanel Access Level:
    Root Administrator
    Twitter:
    Last I looked hostmerit config had a lot of duplicates in it. Not that that matters I don't think but I don't use it. I do use a bunch of other rulesets that need to be tweaked though, as the wrong setup can really spike the server load (and is why I asked about mod_security). A post on that here at gotroot.
    http://tinyurl.com/28pde3
     
  12. noimad1

    noimad1 Well-Known Member

    Joined:
    Mar 27, 2003
    Messages:
    627
    Likes Received:
    0
    Trophy Points:
    16
    OK, it has happened again. I did the netstat -tup but I have no idea what I'm looking at.


    Here are some of the items from that report. I see a lot of http stuff in there, but http is supposedly not running except that one process that is taking up 98% of my cpu that I can't kill.


    tcp 307 0 d1:http 84.238.190.202:4399 CLOSE_WAIT -
    tcp 309 0 d1:http 84.238.190.202:4398 CLOSE_WAIT -
    tcp 305 0 d1:http 84.238.190.202:4397 CLOSE_WAIT -
    tcp 303 0 d1:http 84.238.190.202:4396 CLOSE_WAIT -
    tcp 334 0 d1:http adsl-156-176-72.mia.be:4141 CLOSE_WAIT -
    tcp 562 0 d1:http admin-151-183.potsdam.:2582 CLOSE_WAIT -
    tcp 307 0 d1:http 84.238.190.202:4386 CLOSE_WAIT -
    tcp 309 0 d1:http 84.238.190.202:4385 CLOSE_WAIT -
    tcp 263 0 d1:http CPE00179a496218-CM000:61665 ESTABLISHED -
    tcp 315 0 d1:http 84.238.190.202:4384 CLOSE_WAIT -
    tcp 303 0 d1:http 84.238.190.202:4391 CLOSE_WAIT -
    tcp 0 16 d1:pop3 mobile-032-169-003-18:52384 FIN_WAIT1 -
    tcp 301 0 d1:http 84.238.190.202:4390 CLOSE_WAIT -
    tcp 303 0 d1:http 84.238.190.202:4388 CLOSE_WAIT -
    tcp 0 42 d1:smtp 20178003035.user.velox:3858 ESTABLISHED -
    tcp 326 0 d1:http 124.148.44.200:3823 CLOSE_WAIT -
    tcp 0 0 d1:smtp 85.204.108.180:3573 ESTABLISHED 15363/exim
    tcp 315 0 d1:http 84.238.190.202:4475 CLOSE_WAIT -
    tcp 403 0 d1:http 91.92.234.50:2443 CLOSE_WAIT -
    tcp 0 177 d1:smtp 88.235.237.97:1121 ESTABLISHED 15453/exim
    tcp 340 0 d1:http 77.70.35.3:4694 CLOSE_WAIT -
    tcp 309 0 d1:http 84.238.190.202:4417 CLOSE_WAIT -
    tcp 313 0 d1:http 84.238.190.202:4416 CLOSE_WAIT -
    udp 0 0 localhost:33217 localhost:33212 ESTABLISHED 10025/issCSF
     
Loading...

Share This Page