Monitor server going to power state 4?

rudtek

Member
Jul 19, 2017
22
1
3
Oregon
cPanel Access Level
Root Administrator
I have a VPS server running on Bluehost. The server doesn't have a lot on it, but keeps going down in a power state 4. Bluehost won't provide suggestions on how to monitor to see what is causing the problem. I've tried using top and watching, but is there any way to have a logging to see what is going on when the server stops? I'm not usually on my computer when it happens and so can't find what's going on or how to fix it. Maybe like a performance logger? i'm not even sure what I should be looking for!
 

Jcats

Well-Known Member
PartnerNOC
May 25, 2011
797
152
168
New Jersey
cPanel Access Level
DataCenter Provider
Sounds like a terrific web host! When you say power state are you referring to it going into sleep?
 

rudtek

Member
Jul 19, 2017
22
1
3
Oregon
cPanel Access Level
Root Administrator
they said powerstate 4 meant the server CPU is overwhelmed and so just locks up and my only option is to have them reset. I want to be able what processes / accounts are causing this and they don't know how to do that, because according to them, all the proccess usage data is reset when the server is reset.
 

Jcats

Well-Known Member
PartnerNOC
May 25, 2011
797
152
168
New Jersey
cPanel Access Level
DataCenter Provider
cPanel should send out emails as long as you have the email set in WHM > Basic WebHost Manager Setup

should include ps.txt which will be most useful shows you processes.

did you check dmesg or /var/log/messages for OOM messages?
 

rudtek

Member
Jul 19, 2017
22
1
3
Oregon
cPanel Access Level
Root Administrator
hey Jcats. Thanks for your help. My emails are set up correctly, and I do get system notices. The problem is when the system gets overloaded it just stops, so no emails go out at all. That's why i'm wondering if there is something like a process logger that would show who is using what cpu at the time of system lockup
 

Jcats

Well-Known Member
PartnerNOC
May 25, 2011
797
152
168
New Jersey
cPanel Access Level
DataCenter Provider
Just do a quick Google, example:

How to Get a Email when System Load Average is High ? (Crontab)

then use that on a 1 minute cronjob

If the server locks up with no emails then you may need to monitor in faster intervals in which I would use a 'while' loop with 'sleep' so you can have the script execute as quickly as you want, every second or even every tenth of a second for example.

I can help with that if necessary.

Did you happen to check for the OOM messages by any chance?
 

rudtek

Member
Jul 19, 2017
22
1
3
Oregon
cPanel Access Level
Root Administrator
sorry, I was trying to access /var/logs/messages. Got back up and downloaded the file. Looks like there may be some answers here. maybe my server is being attacked? there are 33000 lines of these errors.

Jun 18 10:41:07 server pam_pwdfile[3655]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:07 server PAM-hulk[3655]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
Jun 18 10:41:09 server pam_pwdfile[3655]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:09 server PAM-hulk[3655]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
Jun 18 10:41:10 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:10 server PAM-hulk[3665]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
Jun 18 10:41:12 server pam_pwdfile[3655]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:12 server PAM-hulk[3655]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
Jun 18 10:41:13 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:13 server PAM-hulk[3665]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
Jun 18 10:41:15 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:15 server PAM-hulk[3665]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
Jun 18 10:41:16 server pam_pwdfile[3690]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:16 server PAM-hulk[3690]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
Jun 18 10:41:19 server pam_pwdfile[3690]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:19 server PAM-hulk[3690]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
Jun 18 10:41:19 server pam_pwdfile[3714]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:19 server PAM-hulk[3714]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
Jun 18 10:41:22 server pam_pwdfile[3714]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:22 server PAM-hulk[3714]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
Jun 18 10:41:23 server pam_pwdfile[3690]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:23 server PAM-hulk[3690]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
Jun 18 10:41:24 server pam_pwdfile[3714]: couldn't open password file /etc/techproxy.shadow
Jun 18 10:41:24 server PAM-hulk[3714]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED

looking in my /etc directory, there is no file there named techproxy.shadow
 

Jcats

Well-Known Member
PartnerNOC
May 25, 2011
797
152
168
New Jersey
cPanel Access Level
DataCenter Provider
I would check:

Code:
/usr/local/cpanel/logs/cphulkd.log
to see what is being hit.

I would also go into:

WHM > cPHulk Brute Force Protection

and check the box:

"Block IP addresses at the firewall level if they trigger brute force protection"

this way as long as the attacks you are getting are not from hundreds/thousands of different IP's your server shouldn't get completely consumed since the firewall will stop the excessive resource usage that is occurring now.
 

Jcats

Well-Known Member
PartnerNOC
May 25, 2011
797
152
168
New Jersey
cPanel Access Level
DataCenter Provider
Just run these via SSH:

Code:
yum -y install bc
Code:
cat > "/root/checkload.sh" <<EOF
#!/bin/bash
 
load=\$(echo \$(cat /proc/loadavg | awk '{print \$1}') \> 3 | bc -l)
if [ "\$load" -ne 0 ]; then
        echo "Your Server Load Alert Needs Attention! " | mail -s "System Load Alert \$load" [email protected]
fi
 
EOF
Code:
chmod +x /root/checkload.sh
Replace the '[email protected]' with your actual email address before you paste the next line:
Code:
sed -i 's/[email protected]/[email protected]/g' /root/checkload.sh
Code:
crontab -l | { cat; echo "* * * * * /root/checkload.sh"; } | crontab -
This will send you an email anytime your server load hits a 1 minute load average of 3 or higher. I'm not sure of your average server load so if its higher than 3 then set 3 higher in the

/root/checkload.sh

script on this line

Code:
load=$(echo $(cat /proc/loadavg | awk '{print $1}') \> 3 | bc -l)
otherwise, the above commands will create the script as well as the cronjob so no need to do anything else.
 
  • Like
Reactions: cPanelLauren

cPanelLauren

Forums Analyst II
Staff member
Nov 14, 2017
8,012
648
263
Houston
cPanel Access Level
DataCenter Provider
Hi @rudtek
I agree with the advice provided by @Jcats. You might also want to look at why this is giving the error it is:

Code:
Jun 18 10:41:15 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow
The file /etc/techproxy.shadow is something added by your hosting provider specifically but it shouldn't be outputting that error all the time.

Once the IP's are added to the firewalls block list you may find that the issue stops.