Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

Monitor server going to power state 4?

Discussion in 'General Discussion' started by rudtek, Jun 17, 2018.

  1. rudtek

    rudtek Member

    Joined:
    Jul 19, 2017
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Oregon
    cPanel Access Level:
    Root Administrator
    I have a VPS server running on Bluehost. The server doesn't have a lot on it, but keeps going down in a power state 4. Bluehost won't provide suggestions on how to monitor to see what is causing the problem. I've tried using top and watching, but is there any way to have a logging to see what is going on when the server stops? I'm not usually on my computer when it happens and so can't find what's going on or how to fix it. Maybe like a performance logger? i'm not even sure what I should be looking for!
     
  2. Jcats

    Jcats Well-Known Member

    Joined:
    May 25, 2011
    Messages:
    720
    Likes Received:
    123
    Trophy Points:
    168
    Location:
    New Jersey
    cPanel Access Level:
    DataCenter Provider
    Sounds like a terrific web host! When you say power state are you referring to it going into sleep?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  3. rudtek

    rudtek Member

    Joined:
    Jul 19, 2017
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Oregon
    cPanel Access Level:
    Root Administrator
    they said powerstate 4 meant the server CPU is overwhelmed and so just locks up and my only option is to have them reset. I want to be able what processes / accounts are causing this and they don't know how to do that, because according to them, all the proccess usage data is reset when the server is reset.
     
  4. Jcats

    Jcats Well-Known Member

    Joined:
    May 25, 2011
    Messages:
    720
    Likes Received:
    123
    Trophy Points:
    168
    Location:
    New Jersey
    cPanel Access Level:
    DataCenter Provider
    cPanel should send out emails as long as you have the email set in WHM > Basic WebHost Manager Setup

    should include ps.txt which will be most useful shows you processes.

    did you check dmesg or /var/log/messages for OOM messages?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  5. rudtek

    rudtek Member

    Joined:
    Jul 19, 2017
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Oregon
    cPanel Access Level:
    Root Administrator
    hey Jcats. Thanks for your help. My emails are set up correctly, and I do get system notices. The problem is when the system gets overloaded it just stops, so no emails go out at all. That's why i'm wondering if there is something like a process logger that would show who is using what cpu at the time of system lockup
     
  6. Jcats

    Jcats Well-Known Member

    Joined:
    May 25, 2011
    Messages:
    720
    Likes Received:
    123
    Trophy Points:
    168
    Location:
    New Jersey
    cPanel Access Level:
    DataCenter Provider
    Just do a quick Google, example:

    How to Get a Email when System Load Average is High ? (Crontab)

    then use that on a 1 minute cronjob

    If the server locks up with no emails then you may need to monitor in faster intervals in which I would use a 'while' loop with 'sleep' so you can have the script execute as quickly as you want, every second or even every tenth of a second for example.

    I can help with that if necessary.

    Did you happen to check for the OOM messages by any chance?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. rudtek

    rudtek Member

    Joined:
    Jul 19, 2017
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Oregon
    cPanel Access Level:
    Root Administrator
    It's weird, I tried accessing that file and the server stopped. I'm resetting again right now.
     
  8. Jcats

    Jcats Well-Known Member

    Joined:
    May 25, 2011
    Messages:
    720
    Likes Received:
    123
    Trophy Points:
    168
    Location:
    New Jersey
    cPanel Access Level:
    DataCenter Provider
    Which file and how did you try and access it? The more details the better :)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. rudtek

    rudtek Member

    Joined:
    Jul 19, 2017
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Oregon
    cPanel Access Level:
    Root Administrator
    sorry, I was trying to access /var/logs/messages. Got back up and downloaded the file. Looks like there may be some answers here. maybe my server is being attacked? there are 33000 lines of these errors.

    Jun 18 10:41:07 server pam_pwdfile[3655]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:07 server PAM-hulk[3655]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
    Jun 18 10:41:09 server pam_pwdfile[3655]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:09 server PAM-hulk[3655]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
    Jun 18 10:41:10 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:10 server PAM-hulk[3665]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
    Jun 18 10:41:12 server pam_pwdfile[3655]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:12 server PAM-hulk[3655]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
    Jun 18 10:41:13 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:13 server PAM-hulk[3665]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
    Jun 18 10:41:15 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:15 server PAM-hulk[3665]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
    Jun 18 10:41:16 server pam_pwdfile[3690]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:16 server PAM-hulk[3690]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
    Jun 18 10:41:19 server pam_pwdfile[3690]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:19 server PAM-hulk[3690]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
    Jun 18 10:41:19 server pam_pwdfile[3714]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:19 server PAM-hulk[3714]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
    Jun 18 10:41:22 server pam_pwdfile[3714]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:22 server PAM-hulk[3714]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
    Jun 18 10:41:23 server pam_pwdfile[3690]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:23 server PAM-hulk[3690]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED
    Jun 18 10:41:24 server pam_pwdfile[3714]: couldn't open password file /etc/techproxy.shadow
    Jun 18 10:41:24 server PAM-hulk[3714]: Brute force detection active: 580 LOGIN DENIED -- EXCESSIVE FAILURES -- IP TEMP BANNED

    looking in my /etc directory, there is no file there named techproxy.shadow
     
  10. Jcats

    Jcats Well-Known Member

    Joined:
    May 25, 2011
    Messages:
    720
    Likes Received:
    123
    Trophy Points:
    168
    Location:
    New Jersey
    cPanel Access Level:
    DataCenter Provider
    I would check:

    Code:
    /usr/local/cpanel/logs/cphulkd.log
    to see what is being hit.

    I would also go into:

    WHM > cPHulk Brute Force Protection

    and check the box:

    "Block IP addresses at the firewall level if they trigger brute force protection"

    this way as long as the attacks you are getting are not from hundreds/thousands of different IP's your server shouldn't get completely consumed since the firewall will stop the excessive resource usage that is occurring now.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  11. rudtek

    rudtek Member

    Joined:
    Jul 19, 2017
    Messages:
    16
    Likes Received:
    0
    Trophy Points:
    1
    Location:
    Oregon
    cPanel Access Level:
    Root Administrator
    okay. I'll try that. i would love help on that script too. It's a bit over my head.
     
  12. Jcats

    Jcats Well-Known Member

    Joined:
    May 25, 2011
    Messages:
    720
    Likes Received:
    123
    Trophy Points:
    168
    Location:
    New Jersey
    cPanel Access Level:
    DataCenter Provider
    Just run these via SSH:

    Code:
    yum -y install bc
    Code:
    cat > "/root/checkload.sh" <<EOF
    #!/bin/bash
     
    load=\$(echo \$(cat /proc/loadavg | awk '{print \$1}') \> 3 | bc -l)
    if [ "\$load" -ne 0 ]; then
            echo "Your Server Load Alert Needs Attention! " | mail -s "System Load Alert \$load" mail@helloacm.com
    fi
     
    EOF
    Code:
    chmod +x /root/checkload.sh
    Replace the 'replace@me.com' with your actual email address before you paste the next line:
    Code:
    sed -i 's/mail@helloacm.com/replace@me.com/g' /root/checkload.sh
    Code:
    crontab -l | { cat; echo "* * * * * /root/checkload.sh"; } | crontab -
    
    This will send you an email anytime your server load hits a 1 minute load average of 3 or higher. I'm not sure of your average server load so if its higher than 3 then set 3 higher in the

    /root/checkload.sh

    script on this line

    Code:
    load=$(echo $(cat /proc/loadavg | awk '{print $1}') \> 3 | bc -l)
    otherwise, the above commands will create the script as well as the cronjob so no need to do anything else.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    cPanelLauren likes this.
  13. cPanelLauren

    cPanelLauren Forums Analyst
    Staff Member

    Joined:
    Nov 14, 2017
    Messages:
    2,721
    Likes Received:
    186
    Trophy Points:
    143
    Location:
    Houston
    cPanel Access Level:
    DataCenter Provider
    Hi @rudtek
    I agree with the advice provided by @Jcats. You might also want to look at why this is giving the error it is:

    Code:
    Jun 18 10:41:15 server pam_pwdfile[3665]: couldn't open password file /etc/techproxy.shadow
    
    The file /etc/techproxy.shadow is something added by your hosting provider specifically but it shouldn't be outputting that error all the time.

    Once the IP's are added to the firewalls block list you may find that the issue stops.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice