VPS intermittently unreachable - hostname unresolvable

[Jon Erickson]

Hi there, hoping someone can help point me in the right direction. I've got a VPS running WHM and intermittently (becoming more often - like once a day) the server hostnames are unresolvable. The server IP is pingable, I've followed the server high load trouble shooting thread which doesn't appear to be an issue - plenty of resources available. DNS has been verified and is working correctly. I am thinking it is an Apache issue as the server is still reachable via SSH using the server hostname, but the websites are not reachable including WHM. What is the next step?

 

[cPanelLauren]

Hi @Jon Erickson


Is anything noted in the logs? You'd want to check the apache error log as well as messages to start:

/etc/apache2/logs/error_log

/var/log/messages

 

[Jon Erickson]

@cPanelLauren nothing of interest in the apache logs.

/var/log/messages does contain a log from the day it went down with 42MB of the following:

Apr 14 03:06:27 vps named[1369]: client 74.63.xx.xxx#19679: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 74.63.17.242#43551: view external: query (cache) 'example.com/NS/IN' denied
Apr 14 03:06:27 vps named[1369]: client 172.68.xx.xxx#35188: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 172.68.xx.xxx#24370: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 172.68.xx.xxx#52052: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 162.158.76.247#43998: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 172.68.xx.xxx#35195: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 172.68.xx.xxx#48101: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 162.158.76.247#33441: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 172.68.xx.xxx#24865: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 162.158.76.247#28850: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:27 vps named[1369]: client 162.158.xx.xx#57938: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:27 vps named[1369]: client 162.158.xx.xx#62880: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:27 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:27 vps named[1369]: client 172.68.xx.xxx#41236: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 172.68.64.167#19989: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 172.68.xx.xxx#38721: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 172.68.64.167#62439: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:27 vps named[1369]: client 172.68.xx.xxx#43362: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:27 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:28 vps named[1369]: client 74.63.xx.xxx#20238: view external: query (cache) 'example.com/NS/IN' denied
Apr 14 03:06:28 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:28 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:28 vps named[1369]: client 74.63.xx.xxx#5173: view external: query (cache) 'example.com/NS/IN' denied
Apr 14 03:06:28 vps named[1369]: client 74.63.xx.xxx#13866: view external: query (cache) 'example.com/NS/IN' denied
Apr 14 03:06:28 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:28 vps named[1369]: client 74.63.xx.xxx#20479: view external: query (cache) 'example.com/NS/IN' denied
Apr 14 03:06:28 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:28 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:28 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:29 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:29 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:29 vps named[1369]: client 74.63.xx.xxx#27014: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:29 vps named[1369]: client 74.63.xx.xxx#7681: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:29 vps named[1369]: client 74.63.xx.xxx#52375: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:29 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:29 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:29 vps named[1369]: client 74.63.xx.xxx#19671: view external: query (cache) 'example.com/AAAA/IN' denied
Apr 14 03:06:29 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:29 vps named[1369]: client 74.125.xx.xx
Apr 14 03:06:31 vps named[1369]: client 172.68.xx.xxx#23655: view external: query (cache) 'example.com/A/IN' denied
Apr 14 03:06:31 vps named[1369]: client 162.158.xx.xxx#12867: view external: query (cache) 'example.com/A/IN' denied
Apr 14 03:06:31 vps named[1369]: client 74.63.xx.xxx#11232: view external: query (cache) 'example.com/A/IN' denied
Apr 14 03:06:31 vps named[1369]: client 172.68.64.215#41898: view external: query (cache) 'example.com/A/IN' denied
Apr 14 03:06:31 vps named[1369]: client 162.158.xx.xxx#34989: view external: query (cache) 'example.com/A/IN' denied
Apr 14 03:06:31 vps named[1369]: client 162.158.xx.xxx#31825: view external: query (cache) 'example.com/A/IN' denied
Apr 14 03:06:32 vps named[1369]: client 172.68.64.215#36428: view external: query (cache) 'example.com/A/IN' denied
Apr 14 03:06:32 vps named[1369]: client 162.158.xx.xxx#55653: view external: query (cache) 'example.com/A/IN' denied
Apr 14 03:06:32 vps named[1369]: client 162.158.xx.xxx#13682: view external: query (cache) 'example.com/A/IN' denied

Super strange, these lines appear almost every second of every day? Is this someone trying to brute force? Its coupled with several lines of:

vps PAM-hulk[19592]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED

Apr  7 07:14:57 vps PAM-hulk[19592]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED

Apr  7 07:15:01 vps PAM-hulk[19592]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED

Apr  7 07:15:08 vps PAM-hulk[19631]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED

Apr  7 07:15:11 vps PAM-hulk[19631]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED

Apr  7 07:15:13 vps PAM-hulk[19631]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED

Also, I might add, I looked at the daily process log, and it also appeared like memcache was using 60% of the CPU when it normally consumes 3-4%. I've switched my services over to Redis to see if it makes a difference and uninstalled memcache.

 

[cPanelLauren]

Hi @Jon Erickson

neither of those look to be related to the issue at hand and the second batch of logs does indicate a potential brute force but all of it was blocked.

I was looking more for logs that correspond to the downtime.