Server slowly grinds to a halt every day or so, but blocking UDP traffic for 5 seconds instantly restores it?

Suburbazine

Registered
Apr 17, 2021
1
0
1
Baton Rouge, LA
cPanel Access Level
Reseller Owner
Hello,

Got a stumper for y'all:


  • CENTOS 7.9 xen hvm [server]
  • V94.0.4
  • Load Averages: 5.12 5.34 3.67
  • 16 core VDS (Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz)
  • 16GB RAM (less than 25% utilized)
  • 900GB disk (60% utilized)
  • ~60 reseller accounts with hosting
  • ~15 reseller accounts with combined hosting and email
System resource usage looks perfectly fine in TOP and I/O monitoring looks fine too.

But at random, the entire WHM instance will grind to a halt and stop serving requests entirely (internally and externally). Won't even throw an error. Logging hasn't indicated what exactly is the issue. Packet capture suggests spurious UDP traffic pointed at customer cPanel accounts from outside world may be the culprit but it's unknown why. What reinforces something UDP based being the culprit is that I can drop a global UDP traffic block on the upstream firewall for just 5 seconds... and the server instantly springs back to life like nothing happened. It'll run for a while and then have a problem again.

Suspicion was something DNS related (WHM using PowerDNS), but I and my team are coming up dry on the "why". Don't think it's a DNS resolution issue... it's a lack of any response whatsoever to even locally served traffic.

Any ideas we should be looking at?
 
Last edited by a moderator:

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
6,810
895
313
cPanel Access Level
Root Administrator
Hey there! If you block UDP traffic that would drop all DNS connections to the machine for a brief second, so any type of attack would likely stop as it would not be able to resolve the hostnames during that time. For the issues you are describing, I would be looking into a DoS attack of some sort:


You can use that article to see if that may be the case.
 
  • Like
Reactions: Suburbazine