Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

p0f failing

Discussion in 'General Discussion' started by Spork Schivago, May 3, 2018.

  1. Spork Schivago

    Spork Schivago Well-Known Member

    Joined:
    Jan 21, 2016
    Messages:
    594
    Likes Received:
    63
    Trophy Points:
    28
    Location:
    corning, ny
    cPanel Access Level:
    Root Administrator
    @cPanelLauren,

    I was unable to figure out how to send you a PM. I even went to your page, but saw no options.

    Is this something I should be worried about?
    Code:
    Service Check Raw Output
    
    (XID 9xtngq) The “p0f” service is down.
    
    The subprocess “/usr/local/cpanel/scripts/restartsrv_p0f” reported error number 3 when it ended.
    
    Startup Log
    
    May 03 05:27:34 franklin.business.net p0f[3800]: [+] Intercepting traffic on interface 'any'.
    May 03 05:27:34 franklin.business.net p0f[3800]: [+] Custom filtering rule enabled: less 400 and not dst port 80 and not dst port 443 and tcp[13] & 8==0
    May 03 05:27:34 franklin.business.net p0f[3800]: [+] Listening on API socket '/var/cpanel/userhomes/cpanelconnecttrack/p0f.socket' (max 20 clients).
    May 03 05:27:34 franklin.business.net p0f[3800]: [+] Privileges dropped: uid 989, gid 986, root '/var/cpanel/userhomes/cpanelconnecttrack'.
    May 03 05:27:34 franklin.business.net p0f[3800]: [+] Daemon process created, PID 3814 (stderr kept as-is).
    May 03 05:27:34 franklin.business.net p0f[3800]: Good luck, you're on your own now!
    May 03 05:27:34 franklin.business.net systemd[1]: Started p0f passive fingerprinter.
    May 03 05:38:14 franklin.business.net p0f[3800]: [!] WARNING: User-initiated shutdown.
    May 03 05:38:14 franklin.business.net systemd[1]: Stopping p0f passive fingerprinter...
    May 03 05:38:14 franklin.business.net systemd[1]: Stopped p0f passive fingerprinter.
    
    Someone's been trying to get into my server for the past few weeks, but they really started kicking it up today / last night. I got a message from cPanel saying cPanel was killing a program because we were almost out of memory. That got me a bit worried. I've been watching them try to get in and they've always been attempting the exact same ports. They get blocked, they reconnect with a different IP, try the exact same ports. Well, after that cPanel message, I log in, and they're not trying the same ports anymore. No, they're trying ports that are pretty dang close to ports that I have open (one off, for instance). And the whole 8==0 doesn't seem right, and Good luck, you're on your own now! That worries me a bit!
     
  2. cPanelLauren

    cPanelLauren Forums Analyst
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    1,855
    Likes Received:
    135
    Trophy Points:
    118
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @Spork Schivago

    As you can see, I moved this to its own thread so it could be addressed.

    For p0f failing that's an interesting error as it indicates that it's a user level stop. if you run the following via CLI what is the output?

    Code:
    grep -i "out of memory" /var/log/messages
    When you try to restart p0f now does it give you the same error? If it does can you show me the version of p0f you're using by running the following command;

    Code:
    rpm -qa |grep p0f
    
    For example my version is as follows:
    Code:
    [root@server ~]# rpm -qa |grep p0f
    p0f-3.09b-1.cp1150.x86_64
    More information on p0f can be found in our documentation here: Service Manager - Version 68 Documentation - cPanel Documentation
    and in theirs here: p0f/p0f
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. Spork Schivago

    Spork Schivago Well-Known Member

    Joined:
    Jan 21, 2016
    Messages:
    594
    Likes Received:
    63
    Trophy Points:
    28
    Location:
    corning, ny
    cPanel Access Level:
    Root Administrator
    I temporarily shut down the server, but it wasn't just p0f that failed. That was the first, then there were messages about cPanel not being able to access 127.0.0.1:8(thousand something), which was the java stuff, and then I get a whooooole bunch of messages like that, all red. So I rebooted. But now I've temporarily shutdown the VPS, hoping after a while, they'll give up. But it makes me think perhaps they found away in through cPanel.

    Even with all the security software I have, it's hard to stop them because they have soooooo many IPs from all around the world. This is why I don't really care for people who pirate software. People think they're getting something for free, but 1) It's stealing, and 2) It's almost always infected with something. They don't notice any symptoms but now-a-days, "hackers", and I use the term very loosely here, I think would rather infect a machine and not be noticed, rather than infect a machine and directly be noticed. They create these large botnets that they can rent out or use for massive DDoS's, etc.
     
  4. Spork Schivago

    Spork Schivago Well-Known Member

    Joined:
    Jan 21, 2016
    Messages:
    594
    Likes Received:
    63
    Trophy Points:
    28
    Location:
    corning, ny
    cPanel Access Level:
    Root Administrator
    Here's some more for you to look at:
    Code:
    The service “tailwatchd” appears to be down.
    Server hostname.business.net
    
    Primary IP <personal.com's IPv4 address, not business.net's IPv4 address>
    Address
    
    Service tailwatchd
    Name
    
    Service failed ⛔
    Status
    
    Notification The service “tailwatchd” appears to be down.
    
    Service
    Check Raw (XID 4cr3ts) The “tailwatchd” service is down.
    Output
    
                 Used        2.29 GB
    Memory       Available   1.56 GB
    Information  Installed   3.85 GB
    
    Load         1.93 0.94 0.36
    Information
    
    Uptime       1 minute and 40 seconds
    
    IOStat       avg-cpu: %user %nice %system %iowait %steal %idle 53.29 0.32 9.37 3.52 0.34 33.16
    Information  Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn loop0 6.89 40.92 7.07 4085 706 sda
                 367.40 16408.60 405.37 1637906 40464 sdb 0.43 11.74 0.00 1172 0
    
              PID     Owner      CPU %   Memory %  Command
    Top       3460    git        38.90    11.71    unicorn master -D -E production -c
    Processes                                      /var/opt/gitlab/gitlab-rails/etc/unicorn.rb
                                                   /opt/gitlab/embedded/service/gitlab-rails/config.ru  
              3152    git        33.90    12.18    sidekiq 5.0.5 gitlab-rails [0 of 25 busy]
              4191    cpanelsolr  9.35     5.93    /usr/lib/jvm/jre-1.8.0/bin/java -server -Xms512m -
                                                   Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -
                                                   XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -
                                                   XX:+UseConcMarkSweepGC -XX:+UseParNewGC -
                                                   XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -
                                                   XX:+CMSScavengeBeforeRemark -
                                                   XX:PretenureSizeThreshold=64m -
                                                   XX:+UseCMSInitiatingOccupancyOnly -
                                                   XX:CMSInitiatingOccupancyFraction=50 -
                                                   XX:CMSMaxAbortablePrecleanTime=6000 -
                                                   XX:+CMSParallelRemarkEnabled -
                                                   XX:+ParallelRefProcEnabled -XX:-
                                                   OmitStackTraceInFastThrow -verbose:gc -
                                                   XX:+PrintHeapAtGC -XX:+PrintGCDetails -
                                                   XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -
                                                   XX:+PrintTenuringDistribution -
                                                   XX:+PrintGCApplicationStoppedTime -
                                                   Xloggc:/home/cpanelsolr/server/logs/solr_gc.log -
                                                   XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -
                                                   XX:GCLogFileSize=20M -
                                                   Dsolr.log.dir=/home/cpanelsolr/server/logs -
                                                   Djetty.port=8984 -DSTOP.PORT=7984 -Dhost=127.0.0.1 -
                                                   Duser.timezone=UTC -
                                                   Djetty.home=/home/cpanelsolr/server -
                                                   Dsolr.solr.home=/home/cpanelsolr/server/solr -
                                                   Dsolr.install.dir=/home/cpanelsolr -Xss256k -
                                                   Dsolr.autoSoftCommit.maxTime=3000 -
                                                   Dsolr.log.muteconsole -
                                                   XX:OnOutOfMemoryError=/home/cpanelsolr/bin/oom_solr.sh 
                                                   8984 /home/cpanelsolr/server/logs -jar start.jar --
                                                   module=http
              4579    root        4.76     0.36    /usr/local/cpanel/scripts/restartsrv_tailwatchd
              4561    root        3.70     0.23    /usr/local/cpanel/bin/tail-check
    
     
  5. cPanelLauren

    cPanelLauren Forums Analyst
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    1,855
    Likes Received:
    135
    Trophy Points:
    118
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @Spork Schivago


    It sounds like the attacker you've been talking about is beginning to cause serious issues with the stability of your server, if there aren't enough resources or if the server is under strain, services will begin to fail. I think in order to address this the situation with the attacker needs to be addressed further. If the attacks are coming from more than one IP address you might want to look at CSF's DDoS features with Connection Tracking which allows ou to set the limit on connections per IP address, there's also the SYNFLOOD protection, a synflood attack is a DoS attack exploiting the TCP connection process itself

    Here are some other links that talk about this:

    Prevent DDOS attack by CSF firewall

    There's also some great info on the CSF forums on how to configure this - their site is down right now but I wouldn't expect it to remain that way for long.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. Spork Schivago

    Spork Schivago Well-Known Member

    Joined:
    Jan 21, 2016
    Messages:
    594
    Likes Received:
    63
    Trophy Points:
    28
    Location:
    corning, ny
    cPanel Access Level:
    Root Administrator
    I do have ConfigServer Firewall already setup and configured. I know understand why snyfloods are and we are protected against them. ConfigServer Firewall is banning them, but the issue is as soon as they get banned, they just use another IP address. This makes it extremely hard. They have a VERY large number of IP addresses.

    ModSec showed them finally trying to access a site on the webserver, /w00tw00t.at.blackhats.romanian.anti-sec

    I believe these are the people responsible for the attack. I believe the way it probably works is something like this.

    They offer a cracked version of some popular software or a keygen for some popular software to download, more than likely from a torrent site. People download it, without realizing it's infected. They don't notice any signs, everything seems to be working fine, and they think oh how, look how smart I am! I got this really expensive software for nothing!

    Then, one day, some "hacker" (using the term loosely here, because to me, I have a different definition of what a hacker is) decides to attack a site like mine, or maybe the Playstation Network. They either rent the botnet that the person who wrote the infected keygen / cracked software from or, they're the people that wrote it themselves, and take control of everyone's PC. They say okay, everyone, start attacking this one site.

    With me, as soon as one IP gets blocked, they just simply use another one. Perhaps ConfigServer Firewall isn't configured 100% properly and was keeping a log of every IP address. Eventually, RAM gets used it, and services start to fail. Hopefully, they've given up their attack by now.

    Gonna fire the VPS back up and see what dmesg shows.
     
  7. cPanelLauren

    cPanelLauren Forums Analyst
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    1,855
    Likes Received:
    135
    Trophy Points:
    118
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    A DDoS attack is exactly what it sounds like is happening - An attack distributed through a collection of IP addresses. That's why I suggested the connection tracking and synflood protection, they should keep too many ip's from connecting on any one port and keep the number of connections from IP down, though admittedly that will not help if they're using more separate IP's than your server can handle.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  8. Spork Schivago

    Spork Schivago Well-Known Member

    Joined:
    Jan 21, 2016
    Messages:
    594
    Likes Received:
    63
    Trophy Points:
    28
    Location:
    corning, ny
    cPanel Access Level:
    Root Administrator
    To me, a DDoS isn't just an attack distributed through a collection of IP addresses, but one that denies regular users access to the site, hence the name Distributed Denial of Service. I'm not denied service to my site, they're just using a distributed collection of IP addresses to try and hack my site. More like a Distributed Hacking of Service or something! {: -)
     
  9. inteldigital

    inteldigital Active Member

    Joined:
    Apr 5, 2018
    Messages:
    38
    Likes Received:
    8
    Trophy Points:
    8
    Location:
    England
    cPanel Access Level:
    Root Administrator
    Twitter:
    Upgrade to 70 and geo-ban IP addresses with more than 5 connections with CSF?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    cPanelLauren and Spork Schivago like this.
  10. Spork Schivago

    Spork Schivago Well-Known Member

    Joined:
    Jan 21, 2016
    Messages:
    594
    Likes Received:
    63
    Trophy Points:
    28
    Location:
    corning, ny
    cPanel Access Level:
    Root Administrator
    I have to wait until 70 hits my current tier. Hopefully it won't be too much longer.
     
  11. inteldigital

    inteldigital Active Member

    Joined:
    Apr 5, 2018
    Messages:
    38
    Likes Received:
    8
    Trophy Points:
    8
    Location:
    England
    cPanel Access Level:
    Root Administrator
    Twitter:
    I'm with you, man. We're waiting for it to land on the release channel before we put it on the production server.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    Spork Schivago likes this.
  12. Spork Schivago

    Spork Schivago Well-Known Member

    Joined:
    Jan 21, 2016
    Messages:
    594
    Likes Received:
    63
    Trophy Points:
    28
    Location:
    corning, ny
    cPanel Access Level:
    Root Administrator
    Yes, for the same reason too it seems. Although the VPS does host a personal domain, it also hosts two business class domains, and even though v70 is close to completion, if we go up a tier, we won't be able to come back down. At that point, we might be stable, for a bit. But then what comes after v70 isn't something that should be ran on a production server.

    In one of my previous jobs, before I was made head of it the IT department, my boss had me write this PHP code and we needed to use a 3rd party module. I suggested one, but he already had one picked out. I told him it was in Alpha testing stage and that wasn't a good idea. That's the reason right there that I was promoted and he was demoted. Over 1/4 of all transactions where just ending up in /dev/null (or the equivalent of whatever the IBM Websphere uses). And we were dealing with very large banks where consumers don't really have accounts, but other banks do. And there where literally millions of transactions a day occuring. It was pretty crazy.
     
    inteldigital likes this.
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice