Please whitelist cPanel in your adblocker so that you’re able to see our version release promotions, thanks!

The Community Forums

Interact with an entire community of cPanel & WHM users!

SOLVED The service “httpd” appears to be down.

Discussion in 'General Discussion' started by LordLiverpool, Nov 10, 2017.

  1. LordLiverpool

    LordLiverpool Well-Known Member

    Joined:
    Dec 27, 2014
    Messages:
    53
    Likes Received:
    9
    Trophy Points:
    8
    cPanel Access Level:
    Root Administrator
    Hello cPanel

    In the past 2 weeks my HTTPD has gone down 3 times.
    Normally it would recover itself.

    Each time I got this email:

    01 httpd appears to be down.JPG

    Looks like the reason it can't recover is:

    "Unable to connect to port 80 on 127.0.0.1: Connection refused: died"

    I looked on the cPanel forum and out on Google for a solution and found similar threads:
    Following the advice contained within them,
    I checked the error log at:

    /usr/local/cpanel/logs/error_log

    And found this at the time it fell over:

    Code:
    [2017-11-09 01:17:12 +0000] info [gather-update-logs] File /usr/local/cpanel/logs/update_analysis/2017-11-09T01:17:07Z.tar.gz has been sent successfully to updatelogdrop.cpanel.net.
    ==> cpsrvd 11.68.0.10 started
    ==> cpsrvd: loading security policy....Done
    ==> cpsrvd: Setting up SSL support ... Done
    ==> cpsrvd: transferred port bindings
    ==> cpsrvd: bound to ports
    ==> cpsrvd 11.68.0.10 started
    ==> cpsrvd: loading security policy....Done
    ==> cpsrvd: Setting up SSL support ... Done
    ==> cpsrvd: transferred port bindings
    ==> cpsrvd: bound to ports
    ==> cpsrvd 11.68.0.10 started
    ==> cpsrvd: loading security policy....Done
    ==> cpsrvd: Setting up SSL support ... Done
    ==> cpsrvd: transferred port bindings
    ==> cpsrvd: bound to ports
    [2017-11-09 01:31:57 +0000] warn [restartsrv_httpd] The 'httpd' service's PID file '/var/run/apache2/httpd.pid' did not appear after 180 seconds.
    
    [2017-11-09 01:31:58 +0000] info [queueprocd] chkservd::Notify Notification => ***********@**mail.*** via EMAIL [eventimportance => High (1)]
    [2017-11-09 01:32:02 +0000] info [cpaddons] Successfully verified signature for cpanel (key types: release).
    [2017-11-09 01:32:04 +0000] info [cpanelsync] Successfully verified signature for cpanel (key types: release).
    [2017-11-09 01:32:04 +0000] info [cpanelsync] Successfully verified signature for cpanel (key types: release).
    [2017-11-09 01:32:05 +0000] info [cpanelsync] Successfully verified signature for cpanel (key types: release).
    [2017-11-09 01:32:05 +0000] info [cpanelsync] Successfully verified signature for cpanel (key types: release).
    [2017-11-09 01:32:12 +0000] info [cpaddons] Successfully verified signature for cpanel (key types: release).
    [2017-11-09 01:36:58 +0000] warn [restartsrv_httpd] The 'httpd' service's PID file '/var/run/apache2/httpd.pid' did not appear after 180 seconds.
    
    I have CSF firewall installed, but I haven't made any changes to it myself for months.

    I can post the iptables rules here if you like? But I noticed they were redacted from other posts.

    I searched the iptable rules for the string "127.0.0.1" but got zero matches. Is that good?

    Search.JPG

    I'm not an expert in server administration so if you want access to my server I can open a support ticket.

    Any help is greatly appreciated.

    Best Regards

    Mike
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  2. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,442
    Likes Received:
    1,962
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello,

    Could you open a support ticket so we can take a closer look?

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    LordLiverpool likes this.
  3. LordLiverpool

    LordLiverpool Well-Known Member

    Joined:
    Dec 27, 2014
    Messages:
    53
    Likes Received:
    9
    Trophy Points:
    8
    cPanel Access Level:
    Root Administrator
    Michael

    I really appreciated you helping me.

    I've opened a ticket: #9019765 and provided a more detailed explanation, as best as I understand.

    Thanks very much.

    Mike
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. LordLiverpool

    LordLiverpool Well-Known Member

    Joined:
    Dec 27, 2014
    Messages:
    53
    Likes Received:
    9
    Trophy Points:
    8
    cPanel Access Level:
    Root Administrator
    OK just to give an update on this thread, so hopefully it helps someone else in the future.

    Apologies in advance to any Level 18 Fire Breathing +5 SysAdmins; this is noob stuff :)

    The cPanel team examined my server and determined that Apache had ran out of Semaphores.

    Each time HTTPD crashed it didn't release the semaphores and eventually my server ran out (I had 128). The cPanel team kindly cleared down the "locked" semaphores as follows:

    Code:
    [19:48:55 mail root@9019765 ~/cptechsjt]cPs# for i in `ipcs | grep admin | awk {'print $2'}`; do ipcrm -s $i; done
    [19:49:06 mail root@9019765 ~/cptechsjt]cPs# /scripts/restartsrv_httpd === 
    And the support analyst advised to read this article:

    Apache: No space left on device: Couldn't create accept lock - major.io

    After reading it, I googled for corroboratory advice and found this article:

    Semaphore limits and many Apache instances on Linux | End Point

    So I decide to increase the number of semaphores from 128 to 256 as follows:


    How many semaphores do I have?

    Code:
    # cat /proc/sys/kernel/sem
    250 32000 32 128
    
    Answer = 128

    Add this line to /etc/sysctl.conf:

    Code:
    kernel.sem = 500 64000 64 256
    
    To make the change take immediate effect, issue this command:

    Code:
    # sysctl -p
    
    Alternatively perform a graceful reboot of your server, (If you can.)

    So how many semaphores do I have now?

    Code:
    # cat /proc/sys/kernel/sem
    500     64000   64      256
    
    You can also check the number of "running" semaphores with this command:

    Code:
    # ipcs -s
    
    I also noticed that one website (wordpress) was creating a disproportionate number of semaphores compared to the other websites (also wordpress) on my server, so it must be a rogue plugin. I will investigate further.

    Thank you to great team at cPanel for helping me out, its genuinely appreciated.

    Best Regards

    Mike
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    cPanelMichael likes this.
  5. oprezy

    oprezy Registered

    Joined:
    Mar 2, 2015
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    cPanel Access Level:
    Website Owner
    Am presently facing similar issues with my web host
     
  6. cPanelMichael

    cPanelMichael Technical Support Community Manager
    Staff Member

    Joined:
    Apr 11, 2011
    Messages:
    45,442
    Likes Received:
    1,962
    Trophy Points:
    363
    cPanel Access Level:
    Root Administrator
    Twitter:
    Hello @oprezy,

    Have you reported the issue to your web hosting provider so they can take a closer look?

    Thank you.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  7. Efren Gutierrez Sosa

    Joined:
    Dec 4, 2018
    Messages:
    1
    Likes Received:
    1
    Trophy Points:
    3
    Location:
    Ecatepec de Morelos
    cPanel Access Level:
    Root Administrator
    Hi!!! i also have this problem y was a really headache. I open a case, however i did´t get a solution, my agent just limit to say that all my services were ok and i needed to hire a system administrator.

    My principal error was that i didn't believed a could be under a DDoS attack, however that was exactly the problem.

    To know who was doing this attack i run this command:

    Code:
    # netstat -an | egrep ':80|:443' | grep ESTABLISHED | awk '{print $5}' | grep -o -E "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort -n | uniq -c | sort -nr
    The first data is the number of current connections, the second is the IP doing GET and POST.

    The mayor number of current connections was 3, however, there was one IP with 279 current connections. First of all, i think the machine has a virus, and i just block connections from this IP using iptables like this:

    Code:
    DROP       all  --  dsl-[removed]anywhere
    Once i did it, after a few minutes my server got normal again, but, that was not the end. The problem came back, it became suspicious because after doing an IP tracker, the location of the attack was the same, so, as the time they were working with a dynamic IP it would be a never end history to try to block every new IP, and my server will be falling every 30 minutes.

    To solve this, i worked again over iptables doing this:

    Code:
    REJECT     tcp  --  anywhere             anywhere             tcp dpt:http flags:FIN,SYN,RST,ACK/SYN #conn src/32 > 10 reject-with tcp-reset
    REJECT     tcp  --  anywhere             anywhere             tcp dpt:https flags:FIN,SYN,RST,ACK/SYN #conn src/32 > 10 reject-with tcp-reset
    REJECT     tcp  --  anywhere             anywhere             tcp dpt:webcache flags:FIN,SYN,RST,ACK/SYN #conn src/32 > 10 reject-with tcp-reset
    REJECT     tcp  --  anywhere             anywhere             tcp dpt:radan-http flags:FIN,SYN,RST,ACK/SYN #conn src/32 > 10 reject-with tcp-reset
    This iptables rules block any new connection from any IP when the current connections are over 10.

    At the time, this has been enough to stop the attack, there are other solutions blocking after "N" seconds, but, that was not my solution because there are many process in my applications that take a lot of time.

    As i understand, this error is not caused in all the cases by the same reason, i expect this could be help to some one with the same
    symptoms.
     
    #7 Efren Gutierrez Sosa, Dec 5, 2018
    Last edited by a moderator: Dec 5, 2018
    cPanelMichael likes this.
Loading...

Share This Page

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice