The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Random High Server Load Spike Issue - WHM - PLZ help!!

Discussion in 'Workarounds and Optimization' started by aae55555, Apr 5, 2012.

  1. aae55555

    aae55555 Active Member

    Joined:
    Apr 5, 2012
    Messages:
    44
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Hi guys

    New to the WHM world - just purchased my first "VM-based" Dedicated Server Solution. I am having a problem with high server loads at random times. Currently my load averages are sitting at: " Load Averages: 0.16 0.14 0.14" , but randomly they will all of a suddent shot up to 17.0, 19.0 etc etc. and the site becomes unaccessible.

    Here are the server specs:

    WHM 11.30.6 (build 7)
    CENTOS 5.8 x86_64 vmware

    Apache 2.2.22
    PHP 5.3.10
    MySQL 5.0.95 (should I upgrade this??)

    CPU: 2 vCPUs
    RAM: 4 GB

    I am running a site with vBulletin Suite (Forums, CMS etc) installed, and receive around 1500/2000 unique hits max on average a day. I was on a Shared hosting package before moving over to the dedicated solution, thinking it would fix the problem, but it seems it is still happening (although less regularly).

    I notice that even in periods of low traffic, the site loads seem to shoot up and it is almost unreachable - takes me around 3 or 4 minutes just to access WHM. I luckily manage to SSH in and reboot the server using "Shutdown -r now" command.

    (I've only noticed it happening around once every week). The backup Cron Jobs as far as I know is set to Weekly, and to execute at 1AM (I think), so I don't think it's that causing it.

    Please tell me what info I can provide so that you can assist me in determining what is causing the high loads. I only have 1 domain/user on the server so that may help us narrow down the possiblities.

    Here is what I see under "Daily Process Log"

    User Domain % CPU % MEM MySQL Processes
    user 1 <name> 5.09 70.69 0.1
    root 0.17 2.65 2.0

    User Domain % CPU ▴ Process

    Top Processes
    user1 <dname> 43.0 /usr/bin/php
    user1 <dname> 23.0 /usr/bin/php
    user1 <dname> 12.7 /usr/bin/php

    I am not sure but it would seems that PHP is causing the high CPU usage is this correct?

    If not, what else can I do to try and investigate? are there any commands I can run via SSH next time my server load is really high (e.g. load at 17.0) to try and find out exactly what is causing the problem?

    Could it be random DDoS's?

    Thank you for any help in advance!
     
    #1 aae55555, Apr 5, 2012
    Last edited: Apr 5, 2012
  2. aae55555

    aae55555 Active Member

    Joined:
    Apr 5, 2012
    Messages:
    44
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
  3. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    If the issue appears to be due to PHP, you might consider optimizing PHP on the machine along with Apache. What is your current PHP handler and PHP version?

    Code:
    php -v
    /usr/local/cpanel/bin/rebuild_phpconf --current
     
  4. aae55555

    aae55555 Active Member

    Joined:
    Apr 5, 2012
    Messages:
    44
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Firstly, thank you so much for the reply!

    Here is the info you requested:

    php -v

    /usr/local/cpanel/bin/rebuild_phpconf --current
    In terms of optimising PHP and Apache - are there any basics that I can look into? I am quite new to this. I found this article through a quick search - : A HOWTO on Optimizing PHP with tips and methodologies

    Thank you once again for your response and look forward to your reply!
     
  5. wrender

    wrender Well-Known Member

    Joined:
    Sep 29, 2007
    Messages:
    69
    Likes Received:
    3
    Trophy Points:
    8
    I have found it very helpful to do a "tail -f /usr/local/apache/domlogs/domainnameinquestion.com". This provides detailed information about what is specifically accessing that domain that is being overloaded. We have had problems in the past with Google bots and other bots overloading servers, as well as just poorly made websites overloading servers, and random attacks.

    Also, you might want to check out Cloud Linux. Cloud Linux helps prevent individual user accounts from overloading your server. This really help us out with performance issues in our virtualized environment where CPU and DISK IO resources are limited, and in cases where it is a poorly made website with PHP errors that are overloading CPU resources.
     
  6. aae55555

    aae55555 Active Member

    Joined:
    Apr 5, 2012
    Messages:
    44
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    I did reply to this post with the info you requested, however as it contained an external link, it has to be approved by a moderator first. Kindly approve it. Thanks.


    Thanks - It's just me on the server, so only 1 user account which helps narrow it down. I just wanted to try and figure out what exactly is causing the overload on the server and try and fix this!

    Thanks for the tail command - what exactly am I looking out for there? I see a lot of IP'ss, User agents, Time stamps, and GET URL's.

    I am running Vbulletin forums, and have added a few of my own static pages to the site - maybe these are causing the overload errors? Not sure, but i'm having trouble trying to figure out exactly which scripts/pages are causing the overload!

    Today I have been getting a lot of MySQL errors too, in particular this one:

    Not quite sure what is causing that either. The server load is fine.

    Thanks guys.
     
  7. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    If a post is pending moderation, someone who works the forum will approve it at some point as we have multiple administrators on the forum. Unfortunately, I was not around working when the post was made. Thanks!

    As for the PHP details provided, I would suggest switching to DSO and using mod_ruid2, which will run all pages as the user including html ones. You'll have a lot easier time tracking down what pages are being accessed and on which accounts. DSO also runs with less memory than FCGI, doesn't have the configuration requirements it does and should be faster.

    Next, I would suggest adding either xCache or EAccelerator for OPCode caching for your PHP pages. You can use OPCode caching with DSO and with FCGI.

    Finally, for the possible MySQL issues, you might be hitting the maximum size of /tmp with temporary MyISAM table data for MySQL. Check for this error in the MySQL error log:

    Code:
    grep -i "incorrect key" /var/lib/mysql/*.err
    If you do see that error, then you might consider changing the tmpdir for MySQL:

    http://forums.cpanel.net/f5/drive-critical-dev-loop0-var-tmp-97-full-190772.html#post799162
     
  8. Secmas

    Secmas Well-Known Member

    Joined:
    Feb 18, 2005
    Messages:
    321
    Likes Received:
    0
    Trophy Points:
    16
    @ Tristan,
    what is best, to run mod_ruid2 with DSO or with SuPHP?

    Regards,

    Sergio
     
  9. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    This is going to be my opinion, so please keep that in mind. My opinion is running DSO with mod_ruid2 would be the best, since to me the best is what provides the following:

    - fastest speeds
    - least memory usage
    - ability to track user processes for Apache
    - ability to use caching
    - ability to limit users (such as RLimit directives)
    - security

    DSO provides all of the above by itself other than tracking user processes and security (since you could run processes as another user or as nobody and have insecure file permissions). mod_ruid2 does track every process as the user even html files and images that run on the account. It's far better than suPHP in that regard, since that only shows PHP processes running as the user. For security, it doesn't restrict file permissions, but you can have lower file permissions like suPHP without requiring 666 or 777 on files or folders, and processes do run as the user who owns the file.

    CGI and suPHP are both incredibly slower than DSO or FCGI. FCGI has very high memory usage and a vast array of configuration options that you normally have to modify to run it optimally. DSO has always been the best PHP handler to use and was only really put into disfavor due to the security implications and the inability to track users.
     
  10. Secmas

    Secmas Well-Known Member

    Joined:
    Feb 18, 2005
    Messages:
    321
    Likes Received:
    0
    Trophy Points:
    16
    Thank you Tristan!

    Now, I have been using suPHP for security reasons, if I change to DSO is there something that I could check that suPHP could left behind?
     
  11. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    I'm uncertain the question about "left behind" in regards to suPHP. What precisely are you asking about suPHP versus DSO in regards to security?
     
  12. aae55555

    aae55555 Active Member

    Joined:
    Apr 5, 2012
    Messages:
    44
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Thanks for this. My server just overloaded again, and I am trying to determine what the problem is by using this command.\

    Here is an example of what I see when I run the command

    Not quite sure what I am looking for :(

    Are there any other commands that I can use in cases where my server is overloading and I want to try and find out the cause? e.g. If it is a DDoS, high process usage, bad spider bot/spoofed spider bot overloading the server etc.

    I have tried running the following commands right now while my server is overloaded:

    Thanks for all the other help so far Tristan. I will be implemeting your suggestions soon.
     
    #12 aae55555, Apr 14, 2012
    Last edited: Apr 14, 2012
  13. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Your machine has 0 free swap as all of it is being used per the free -m output. Something is using all the swap. Did you check for "Incorrect key" in the MySQL error logs as previously mentioned? As for a possible DoS, you can check for an Apache-based on using these recommendations:

    http://forums.cpanel.net/f5/how-restart-apache-trough-cpanel-command-line-187252.html#post766101

    You'd also probably want to check /var/log/messages for any possible disk issues. High I/O wait times, such as you are showing at 25% and 100% swap usage could indicate disk problems. Check during the times of the high loads to see if you are showing segfaults in /var/log/messages or anything concerning in /var/log/dmesg logs.
     
  14. aae55555

    aae55555 Active Member

    Joined:
    Apr 5, 2012
    Messages:
    44
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Thanks Tristan, cannot tell you how much I appreciate your responses!

    I did check for incorrect key by running the following command in SSH as you suggested:
    It just returned me to the command/console input which I assume means that nothing was found. Furthermore, if that was the case, I should have been getting MYSQL errors as well right? Well I received no MYSQL errors this time when the server was overloading/freezing up again.

    So now to investigate what is using up the swap.. hmm..

    Could it be a MYSQL setting or other software swap setting that is set up to use too much swap/more than I actually have? Or can I increase the Swap file? I was under the impression that the Swap file is stored on the HDD so I should be able to increase it provided I have enough HDD space (which I do)?

    With regards to the /var/log/messages - thanks for that. Seems it is confirming the swap file is the issue (and also some "out of memory" messages too. Not sure if that is referring to swap or memory).

    Here is an extract from the messages file at the time of the overloading:

    Regarding /var/log/dmesg, I did not find anything suspicious there, however here is that file's contents:

    Thanks once again
     
  15. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    At this point, could you have your datacenter or NOC step in to help? You might be having issues with temperatures on the machine based on what you've provided. They'd be better able to tell you if that's the case.
     
  16. aae55555

    aae55555 Active Member

    Joined:
    Apr 5, 2012
    Messages:
    44
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    I will give my hosting company a ring just now and ask them to check. Do I quote them the info from that "/var/log/messages" file where it says CPU hot 1?

    Should it make any difference it is running in a VM? My hosting company calls this "Cloud Server Hosting" package. Like a dedicated server, but in a VM.

    Also, would I be able to use a tool such as "lm_sensor" to monitor CPU temps? http://www.cyberciti.biz/faq/howto-linux-get-sensors-information/

    Thanks!
     
    #16 aae55555, Apr 16, 2012
    Last edited: Apr 16, 2012
  17. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    I can't say what might be useful for them as I really think they need to log onto the machine and look it over for the messages. Just let them know about the swap issue and high I/O wait times and mention that /var/log/messages has concerning messages that might indicate high temperatures. Any other questions about how to use tools on it would be best directed to them. I don't know the precise setup they have for the virtual node.
     
  18. aae55555

    aae55555 Active Member

    Joined:
    Apr 5, 2012
    Messages:
    44
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Much appreciated. I will let them know and update here
     
  19. aae55555

    aae55555 Active Member

    Joined:
    Apr 5, 2012
    Messages:
    44
    Likes Received:
    0
    Trophy Points:
    6
    cPanel Access Level:
    Root Administrator
    Ok here is the response I got from my hosting provider after notifying them of my concerns, providing them with the free-m output, as well as the concering /var/log/messages output that I posted before.

    I told them I was concerned with overheating CPU, High I/O wait times, and the 100% swap file usage.

    Here is their response:

    What are your thoughts regarding this? I'm not satisfied personally.. They totally ignored my concerns regarding hot CPU and High I/O wait times.

    Thanks.
     
  20. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,623
    Likes Received:
    21
    Trophy Points:
    38
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    Please contact them again to ask them to provide information about temperatures during the times in question, since they didn't reply about that specific part. You might want to provide them with access this time for them to check it as well.
     
Loading...

Share This Page