Spork Schivago

Well-Known Member
Jan 21, 2016
597
64
28
corning, ny
cPanel Access Level
Root Administrator
@cPanelLauren,

I was unable to figure out how to send you a PM. I even went to your page, but saw no options.

Is this something I should be worried about?
Code:
Service Check Raw Output

(XID 9xtngq) The “p0f” service is down.

The subprocess “/usr/local/cpanel/scripts/restartsrv_p0f” reported error number 3 when it ended.

Startup Log

May 03 05:27:34 franklin.business.net p0f[3800]: [+] Intercepting traffic on interface 'any'.
May 03 05:27:34 franklin.business.net p0f[3800]: [+] Custom filtering rule enabled: less 400 and not dst port 80 and not dst port 443 and tcp[13] & 8==0
May 03 05:27:34 franklin.business.net p0f[3800]: [+] Listening on API socket '/var/cpanel/userhomes/cpanelconnecttrack/p0f.socket' (max 20 clients).
May 03 05:27:34 franklin.business.net p0f[3800]: [+] Privileges dropped: uid 989, gid 986, root '/var/cpanel/userhomes/cpanelconnecttrack'.
May 03 05:27:34 franklin.business.net p0f[3800]: [+] Daemon process created, PID 3814 (stderr kept as-is).
May 03 05:27:34 franklin.business.net p0f[3800]: Good luck, you're on your own now!
May 03 05:27:34 franklin.business.net systemd[1]: Started p0f passive fingerprinter.
May 03 05:38:14 franklin.business.net p0f[3800]: [!] WARNING: User-initiated shutdown.
May 03 05:38:14 franklin.business.net systemd[1]: Stopping p0f passive fingerprinter...
May 03 05:38:14 franklin.business.net systemd[1]: Stopped p0f passive fingerprinter.
Someone's been trying to get into my server for the past few weeks, but they really started kicking it up today / last night. I got a message from cPanel saying cPanel was killing a program because we were almost out of memory. That got me a bit worried. I've been watching them try to get in and they've always been attempting the exact same ports. They get blocked, they reconnect with a different IP, try the exact same ports. Well, after that cPanel message, I log in, and they're not trying the same ports anymore. No, they're trying ports that are pretty dang close to ports that I have open (one off, for instance). And the whole 8==0 doesn't seem right, and Good luck, you're on your own now! That worries me a bit!
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,262
313
Houston
Hi @Spork Schivago

As you can see, I moved this to its own thread so it could be addressed.

For p0f failing that's an interesting error as it indicates that it's a user level stop. if you run the following via CLI what is the output?

Code:
grep -i "out of memory" /var/log/messages
When you try to restart p0f now does it give you the same error? If it does can you show me the version of p0f you're using by running the following command;

Code:
rpm -qa |grep p0f
For example my version is as follows:
Code:
[[email protected] ~]# rpm -qa |grep p0f
p0f-3.09b-1.cp1150.x86_64
More information on p0f can be found in our documentation here: Service Manager - Version 68 Documentation - cPanel Documentation
and in theirs here: p0f/p0f
 

Spork Schivago

Well-Known Member
Jan 21, 2016
597
64
28
corning, ny
cPanel Access Level
Root Administrator
I temporarily shut down the server, but it wasn't just p0f that failed. That was the first, then there were messages about cPanel not being able to access 127.0.0.1:8(thousand something), which was the java stuff, and then I get a whooooole bunch of messages like that, all red. So I rebooted. But now I've temporarily shutdown the VPS, hoping after a while, they'll give up. But it makes me think perhaps they found away in through cPanel.

Even with all the security software I have, it's hard to stop them because they have soooooo many IPs from all around the world. This is why I don't really care for people who pirate software. People think they're getting something for free, but 1) It's stealing, and 2) It's almost always infected with something. They don't notice any symptoms but now-a-days, "hackers", and I use the term very loosely here, I think would rather infect a machine and not be noticed, rather than infect a machine and directly be noticed. They create these large botnets that they can rent out or use for massive DDoS's, etc.
 

Spork Schivago

Well-Known Member
Jan 21, 2016
597
64
28
corning, ny
cPanel Access Level
Root Administrator
Here's some more for you to look at:
Code:
The service “tailwatchd” appears to be down.
Server hostname.business.net

Primary IP <personal.com's IPv4 address, not business.net's IPv4 address>
Address

Service tailwatchd
Name

Service failed ⛔
Status

Notification The service “tailwatchd” appears to be down.

Service
Check Raw (XID 4cr3ts) The “tailwatchd” service is down.
Output

             Used        2.29 GB
Memory       Available   1.56 GB
Information  Installed   3.85 GB

Load         1.93 0.94 0.36
Information

Uptime       1 minute and 40 seconds

IOStat       avg-cpu: %user %nice %system %iowait %steal %idle 53.29 0.32 9.37 3.52 0.34 33.16
Information  Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn loop0 6.89 40.92 7.07 4085 706 sda
             367.40 16408.60 405.37 1637906 40464 sdb 0.43 11.74 0.00 1172 0

          PID     Owner      CPU %   Memory %  Command
Top       3460    git        38.90    11.71    unicorn master -D -E production -c
Processes                                      /var/opt/gitlab/gitlab-rails/etc/unicorn.rb
                                               /opt/gitlab/embedded/service/gitlab-rails/config.ru  
          3152    git        33.90    12.18    sidekiq 5.0.5 gitlab-rails [0 of 25 busy]
          4191    cpanelsolr  9.35     5.93    /usr/lib/jvm/jre-1.8.0/bin/java -server -Xms512m -
                                               Xmx512m -XX:NewRatio=3 -XX:SurvivorRatio=4 -
                                               XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8 -
                                               XX:+UseConcMarkSweepGC -XX:+UseParNewGC -
                                               XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 -
                                               XX:+CMSScavengeBeforeRemark -
                                               XX:PretenureSizeThreshold=64m -
                                               XX:+UseCMSInitiatingOccupancyOnly -
                                               XX:CMSInitiatingOccupancyFraction=50 -
                                               XX:CMSMaxAbortablePrecleanTime=6000 -
                                               XX:+CMSParallelRemarkEnabled -
                                               XX:+ParallelRefProcEnabled -XX:-
                                               OmitStackTraceInFastThrow -verbose:gc -
                                               XX:+PrintHeapAtGC -XX:+PrintGCDetails -
                                               XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -
                                               XX:+PrintTenuringDistribution -
                                               XX:+PrintGCApplicationStoppedTime -
                                               Xloggc:/home/cpanelsolr/server/logs/solr_gc.log -
                                               XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=9 -
                                               XX:GCLogFileSize=20M -
                                               Dsolr.log.dir=/home/cpanelsolr/server/logs -
                                               Djetty.port=8984 -DSTOP.PORT=7984 -Dhost=127.0.0.1 -
                                               Duser.timezone=UTC -
                                               Djetty.home=/home/cpanelsolr/server -
                                               Dsolr.solr.home=/home/cpanelsolr/server/solr -
                                               Dsolr.install.dir=/home/cpanelsolr -Xss256k -
                                               Dsolr.autoSoftCommit.maxTime=3000 -
                                               Dsolr.log.muteconsole -
                                               XX:OnOutOfMemoryError=/home/cpanelsolr/bin/oom_solr.sh 
                                               8984 /home/cpanelsolr/server/logs -jar start.jar --
                                               module=http
          4579    root        4.76     0.36    /usr/local/cpanel/scripts/restartsrv_tailwatchd
          4561    root        3.70     0.23    /usr/local/cpanel/bin/tail-check
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,262
313
Houston
Hi @Spork Schivago


It sounds like the attacker you've been talking about is beginning to cause serious issues with the stability of your server, if there aren't enough resources or if the server is under strain, services will begin to fail. I think in order to address this the situation with the attacker needs to be addressed further. If the attacks are coming from more than one IP address you might want to look at CSF's DDoS features with Connection Tracking which allows ou to set the limit on connections per IP address, there's also the SYNFLOOD protection, a synflood attack is a DoS attack exploiting the TCP connection process itself

Here are some other links that talk about this:

Prevent DDOS attack by CSF firewall

There's also some great info on the CSF forums on how to configure this - their site is down right now but I wouldn't expect it to remain that way for long.
 

Spork Schivago

Well-Known Member
Jan 21, 2016
597
64
28
corning, ny
cPanel Access Level
Root Administrator
I do have ConfigServer Firewall already setup and configured. I know understand why snyfloods are and we are protected against them. ConfigServer Firewall is banning them, but the issue is as soon as they get banned, they just use another IP address. This makes it extremely hard. They have a VERY large number of IP addresses.

ModSec showed them finally trying to access a site on the webserver, /w00tw00t.at.blackhats.romanian.anti-sec

I believe these are the people responsible for the attack. I believe the way it probably works is something like this.

They offer a cracked version of some popular software or a keygen for some popular software to download, more than likely from a torrent site. People download it, without realizing it's infected. They don't notice any signs, everything seems to be working fine, and they think oh how, look how smart I am! I got this really expensive software for nothing!

Then, one day, some "hacker" (using the term loosely here, because to me, I have a different definition of what a hacker is) decides to attack a site like mine, or maybe the Playstation Network. They either rent the botnet that the person who wrote the infected keygen / cracked software from or, they're the people that wrote it themselves, and take control of everyone's PC. They say okay, everyone, start attacking this one site.

With me, as soon as one IP gets blocked, they just simply use another one. Perhaps ConfigServer Firewall isn't configured 100% properly and was keeping a log of every IP address. Eventually, RAM gets used it, and services start to fail. Hopefully, they've given up their attack by now.

Gonna fire the VPS back up and see what dmesg shows.
 

cPanelLauren

Product Owner
Staff member
Nov 14, 2017
13,296
1,262
313
Houston
A DDoS attack is exactly what it sounds like is happening - An attack distributed through a collection of IP addresses. That's why I suggested the connection tracking and synflood protection, they should keep too many ip's from connecting on any one port and keep the number of connections from IP down, though admittedly that will not help if they're using more separate IP's than your server can handle.
 

Spork Schivago

Well-Known Member
Jan 21, 2016
597
64
28
corning, ny
cPanel Access Level
Root Administrator
A DDoS attack is exactly what it sounds like is happening - An attack distributed through a collection of IP addresses. That's why I suggested the connection tracking and synflood protection, they should keep too many ip's from connecting on any one port and keep the number of connections from IP down, though admittedly that will not help if they're using more separate IP's than your server can handle.
To me, a DDoS isn't just an attack distributed through a collection of IP addresses, but one that denies regular users access to the site, hence the name Distributed Denial of Service. I'm not denied service to my site, they're just using a distributed collection of IP addresses to try and hack my site. More like a Distributed Hacking of Service or something! {: -)
 

Spork Schivago

Well-Known Member
Jan 21, 2016
597
64
28
corning, ny
cPanel Access Level
Root Administrator
I'm with you, man. We're waiting for it to land on the release channel before we put it on the production server.
Yes, for the same reason too it seems. Although the VPS does host a personal domain, it also hosts two business class domains, and even though v70 is close to completion, if we go up a tier, we won't be able to come back down. At that point, we might be stable, for a bit. But then what comes after v70 isn't something that should be ran on a production server.

In one of my previous jobs, before I was made head of it the IT department, my boss had me write this PHP code and we needed to use a 3rd party module. I suggested one, but he already had one picked out. I told him it was in Alpha testing stage and that wasn't a good idea. That's the reason right there that I was promoted and he was demoted. Over 1/4 of all transactions where just ending up in /dev/null (or the equivalent of whatever the IBM Websphere uses). And we were dealing with very large banks where consumers don't really have accounts, but other banks do. And there where literally millions of transactions a day occuring. It was pretty crazy.
 
  • Like
Reactions: inteldigital