Server slowly grinds to a halt every day or so, but blocking UDP traffic for 5 seconds instantly restores it?

[Suburbazine]

Hello,

Got a stumper for y'all:

• CENTOS 7.9 xen hvm [server]
• V94.0.4
• Load Averages: 5.12 5.34 3.67
• 16 core VDS (Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz)
• 16GB RAM (less than 25% utilized)
• 900GB disk (60% utilized)
• ~60 reseller accounts with hosting
• ~15 reseller accounts with combined hosting and email

System resource usage looks perfectly fine in TOP and I/O monitoring looks fine too.

But at random, the entire WHM instance will grind to a halt and stop serving requests entirely (internally and externally). Won't even throw an error. Logging hasn't indicated what exactly is the issue. Packet capture suggests spurious UDP traffic pointed at customer cPanel accounts from outside world may be the culprit but it's unknown why. What reinforces something UDP based being the culprit is that I can drop a global UDP traffic block on the upstream firewall for just 5 seconds... and the server instantly springs back to life like nothing happened. It'll run for a while and then have a problem again.

Suspicion was something DNS related (WHM using PowerDNS), but I and my team are coming up dry on the "why". Don't think it's a DNS resolution issue... it's a lack of any response whatsoever to even locally served traffic.

Any ideas we should be looking at?

 

[cPRex]

Hey there! If you block UDP traffic that would drop all DNS connections to the machine for a brief second, so any type of attack would likely stop as it would not be able to resolve the hostnames during that time. For the issues you are describing, I would be looking into a DoS attack of some sort:

How can I tell if Apache is experiencing a DDoS attack?

Question How can I tell if Apache is experiencing a DDoS (Distributed Denial-of-Service) attack? Answer If Apache is experiencing a DDoS attack, you may notice that your server's Web sites are ti...

support.cpanel.net

You can use that article to see if that may be the case.