The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Server Going down frequently

Discussion in 'General Discussion' started by ramindia, Dec 20, 2011.

  1. ramindia

    ramindia Well-Known Member

    Joined:
    Apr 3, 2011
    Messages:
    71
    Likes Received:
    0
    Trophy Points:
    56
    Hi all

    I have dedicated server

    every day the server goes down, we are not able to access ssh and WHM pannel
    and server not serving any of the sites hosted in the server

    i have made script to record how many hits coming to the server
    so i got a report before server not accessble around 990

    i have csf and lfd running, and i have set CT_LIMIT = 30

    but still iam getting lot of hits from different servers, how can i block them

    and give assurance to my clients all sites running smooth

    any other suggestions to make server stable.

    when i lost connection to all the access, i need to go to APC panel to restart server.



    Ram
     
    #1 ramindia, Dec 20, 2011
  2. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,622
    Likes Received:
    26
    Trophy Points:
    238
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    I would suggest adding sys-snap.sh and reviewing the processes, memory, and so on when the server goes down:

    http://forums.cpanel.net/f5/server-gets-unreachable-after-swap-full-since-last-update-246452.html#post1019842

    Additionally, you may want to check /var/log/messages during the time the machine is not functional to see what it shows for process(es). If any kernel issues are happening, you'll see those logged to /var/log/messages location.

    After that, try checking sar output to see if you might be running out of memory or have high I/O

    You should be able to get an idea of the time(s) of the occurrence using "last | grep boot" in command line, since that will show each reboot on the machine.
     
    #2 cPanelTristan, Dec 20, 2011
  3. ramindia

    ramindia Well-Known Member

    Joined:
    Apr 3, 2011
    Messages:
    71
    Likes Received:
    0
    Trophy Points:
    56
    Hi

    thanks

    since 24 hours the server got hanged

    i am running sys snap, waiting for the server halt to take output of sys snaps

    when the machine is not functioning, iam not able to access, but later when i look at /var/log/message
    i see maximum hits to http and load high on 287 * on 4 cpus and memory also using 80% of 4GBRAM

    iam looking sar report if i can see any clue why the server going down frequently

    last | grep boot
    reboot system boot 2.6.18-194.26.1. Tue Dec 20 06:27 (17:09)
    reboot system boot 2.6.18-194.26.1. Thu Dec 15 06:06 (5+17:30)
    reboot system boot 2.6.18-194.26.1. Sun Dec 11 23:38 (8+23:57)
    reboot system boot 2.6.18-194.26.1. Sun Dec 11 05:52 (9+17:44)
    reboot system boot 2.6.18-194.26.1. Mon Dec 5 00:27 (15+23:09)
    reboot system boot 2.6.18-194.26.1. Sat Dec 3 00:01 (17+23:35)
    reboot system boot 2.6.18-194.26.1. Fri Dec 2 01:16 (18+22:20)
    reboot system boot 2.6.18-194.26.1. Fri Nov 11 14:37 (39+08:58)
    reboot system boot 2.6.18-194.26.1. Fri Nov 11 14:57 (00:-20)
    reboot system boot 2.6.18-194.26.1. Mon Nov 7 00:19 (4+14:17)
    reboot system boot 2.6.18-194.26.1. Wed Nov 2 01:04 (9+14:32)
    reboot system boot 2.6.18-194.26.1. Tue Nov 1 01:43 (10+13:53)
    reboot system boot 2.6.18-194.26.1. Thu Oct 27 01:19 (5+00:21)
    reboot system boot 2.6.18-194.26.1. Tue Oct 25 07:49 (6+17:51)
    reboot system boot 2.6.18-194.26.1. Mon Oct 24 07:07 (7+18:34)
    reboot system boot 2.6.18-194.26.1. Thu Oct 20 10:17 (11+15:24)
    reboot system boot 2.6.18-194.26.1. Thu Oct 20 10:13 (00:00)
    reboot system boot 2.6.18-194.26.1. Fri Sep 23 02:57 (27+07:16)
    reboot system boot 2.6.18-194.26.1. Fri Sep 23 02:54 (00:00)
    reboot system boot 2.6.18-194.26.1. Thu Sep 22 08:42 (17:58)
    reboot system boot 2.6.18-194.26.1. Wed Feb 23 14:17 (211+11:24)
    reboot system boot 2.6.18-194.26.1. Tue Dec 28 01:02 (57+12:47)
    reboot system boot 2.6.18-194.26.1. Mon Dec 20 14:38 (64+23:11)
    reboot system boot 2.6.18-194.26.1. Mon Dec 20 22:55 (-8:-18)
    reboot system boot 2.6.18-194.26.1. Thu Dec 16 00:42 (4+13:54)
    reboot system boot 2.6.18-194.26.1. Thu Dec 16 00:31 (4+14:05)
    reboot system boot 2.6.18-194.26.1. Thu Dec 16 00:24 (00:01)
    reboot system boot 2.6.18-194.26.1. Thu Dec 16 00:20 (00:02)
    reboot system boot 2.6.18-194.26.1. Thu Dec 16 00:09 (00:08)



    Support
    Main >> Security Center >> Scan for Trojan Horses
    Scan for Trojan Horses
    Appears Clean

    /dev/core
    /dev/stderr

    Scanning for Trojan Horses.....
    .

    Possible Trojan - /usr/sbin/pureauth
    .
    .
    .
    .

    Possible Trojan - /etc/cron.daily/logrotate
    .

    Possible Trojan - /usr/bin/cpan
    .
    .

    Possible Trojan - /usr/bin/instmodsh
    .

    Possible Trojan - /usr/bin/prove
    .

    Possible Trojan - /usr/bin/xsubpp
    .
    .
    .
    6 POSSIBLE Trojans Detected


    any suggestions

    Ram
     
    #3 ramindia, Dec 20, 2011
  4. faisikhan

    faisikhan Well-Known Member

    Joined:
    Dec 12, 2011
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    56
    Location:
    Islamabad, Pakistan
    cPanel Access Level:
    Root Administrator
    Hi,

    1. First of all "Server Reboot" isn't the permanent solution of any tiny or big issue, you may get rid of the issue at run time just rebooting the server but you won't get rid of it permanently.
    2. So you have to make your server secure to avoid unwanted hits because there may be some security hole so follow that link 20 Linux Server Hardening Security Tips
    3. As cPanelTristan said tail -f /var/log/messages will help you a lot to get the root cause of the unwanted hits coming from different IP addresses & so block those IPs.
    4. Run top command to see the process which is consuming much memory and cpu resources and restart it once before you reboot the server like httpd in your case.
    5. Check the available memory upon the server and extend it if possible or remove swapping processes Find out what is using your swap | All things Sysadmin
     
    #4 faisikhan, Dec 21, 2011
  5. ramindia

    ramindia Well-Known Member

    Joined:
    Apr 3, 2011
    Messages:
    71
    Likes Received:
    0
    Trophy Points:
    56
    Hi faiskhan

    thanks for quick reply and advice

    1. iam rebooting the server only when iam not able to access the server for several hours.

    2. most of the packages are removed, if they are not using.

    3. iam getting some different IP at a time like 900hits, that time iam not able to access the server at all see who is hitting me, so i have added in LFD CT_LIMIT=30 to block. still the server unreachable some time ( so i force to reboot, if i get access sure iam doing restart http, so i see the load go down fast.)

    4. top -c always show more process running by httpd, thats due to attacks

    5. Swap is always free only, but i see sometime memory occupied.

    i will try some of your tips.

    i welcome any more suggestion to address my problem to fix permanently.

    thanks

    Ram
     
    #5 ramindia, Dec 21, 2011
  6. cPanelTristan

    cPanelTristan Quality Assurance Analyst
    Staff Member

    Joined:
    Oct 2, 2010
    Messages:
    7,622
    Likes Received:
    26
    Trophy Points:
    238
    Location:
    somewhere over the rainbow
    cPanel Access Level:
    Root Administrator
    #6 cPanelTristan, Dec 25, 2011
  7. ramindia

    ramindia Well-Known Member

    Joined:
    Apr 3, 2011
    Messages:
    71
    Likes Received:
    0
    Trophy Points:
    56
    Hi cPanelTristan

    yes i see SYN Attacks

    [error] server reached MaxClients setting, consider raising the MaxClients setting

    the site normally load was not more than 100connections
    but i have set to 512 connections

    when the last flood happens it went to 1024 connections

    I have enabled CT_LIMIT=30

    I have enabled system-snapshot

    before server stopped, i took the log, below one ( IP and other information replaced due to security problems)


    File attached (cpanel-forum.txt)


    After i have hard rebooted the server

    I have enabled the SYN Flood config in CFD/LFD

    SYNFLOOD = "1"
    SYNFLOOD_RATE = "100/s"
    SYNFLOOD_BURST = "150"

    CONNLIMIT = "80;10"


    Iam monitoring the server

    any other suggestion, what iam missing

    Ram
     

    Attached Files:

    #7 ramindia, Dec 26, 2011
  8. faisikhan

    faisikhan Well-Known Member

    Joined:
    Dec 12, 2011
    Messages:
    88
    Likes Received:
    0
    Trophy Points:
    56
    Location:
    Islamabad, Pakistan
    cPanel Access Level:
    Root Administrator
    Hi

    It is good that you are monitoring the server very closely & from the file attached I have seen that the problem is caused by the unwanted IP hits causing the server down and consuming other resources. So I hope so to block those IP addresses & filtering the network traffic(via http or https) will prevent the server going down. Did you define any IP-tables rule for that before, if not see here for filtering the network traffic /http://www.thegeekstuff.com/2011/06/iptables-rules-examples/ , also see No.23 for DoS attack.
     
    #8 faisikhan, Dec 27, 2011
  9. ramindia

    ramindia Well-Known Member

    Joined:
    Apr 3, 2011
    Messages:
    71
    Likes Received:
    0
    Trophy Points:
    56
    Hi faiskhan

    i have added

    I have enabled the SYN Flood config in CFD/LFD

    SYNFLOOD = "1"
    SYNFLOOD_RATE = "100/s"
    SYNFLOOD_BURST = "150"

    CONNLIMIT = "80;10"

    Looks server stable past 4days

    and server still under monitoring 24hours basis
    to see if any problem repeats again

    Ram
     
    #9 ramindia, Dec 31, 2011
  10. ruzbehraja

    ruzbehraja Well-Known Member

    Joined:
    May 19, 2011
    Messages:
    392
    Likes Received:
    11
    Trophy Points:
    68
    cPanel Access Level:
    Root Administrator
    • How many sites are hosted on that server?
    • Is the attack on a particular site(s)?
    • Do you have mod_security installed and enabled?
     
    #10 ruzbehraja, Jan 1, 2012
  11. ramindia

    ramindia Well-Known Member

    Joined:
    Apr 3, 2011
    Messages:
    71
    Likes Received:
    0
    Trophy Points:
    56
    Hi

    2 sites are hosted on that server

    only 1 domain getting attacked

    iam not sure "Do you have mod_security installed and enabled?"

    Ram
     
    #11 ramindia, Jan 1, 2012
  12. ruzbehraja

    ruzbehraja Well-Known Member

    Joined:
    May 19, 2011
    Messages:
    392
    Likes Received:
    11
    Trophy Points:
    68
    cPanel Access Level:
    Root Administrator
    #12 ruzbehraja, Jan 2, 2012
  13. ramindia

    ramindia Well-Known Member

    Joined:
    Apr 3, 2011
    Messages:
    71
    Likes Received:
    0
    Trophy Points:
    56
    Hi

    thanks for the suggestion

    as per the document here is my config


    SYN:


    more /proc/sys/net/ipv4/tcp_synack_retries
    5

    more /proc/sys/net/ipv4/tcp_max_syn_backlog
    1024
    more /proc/sys/net/ipv4/tcp_syncookies
    1

    Spoofing :

    more /proc/sys/net/ipv4/conf/*/rp_*
    ::::::::::::::
    /proc/sys/net/ipv4/conf/all/rp_filter
    ::::::::::::::
    0
    ::::::::::::::
    /proc/sys/net/ipv4/conf/default/rp_filter
    ::::::::::::::
    1
    ::::::::::::::
    /proc/sys/net/ipv4/conf/eth0/rp_filter
    ::::::::::::::
    1
    ::::::::::::::
    /proc/sys/net/ipv4/conf/lo/rp_filter
    ::::::::::::::
    0



    iptables rules


    16 CONNLIMIT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 flags:0x17/0x02 #conn/32 > 10



    Chain SYNFLOOD (1 references)
    num target prot opt source destination
    1 RETURN all -- 0.0.0.0/0 0.0.0.0/0 limit: avg 100/sec burst 150
    2 LOG all -- 0.0.0.0/0 0.0.0.0/0 limit: avg 30/min burst 5 LOG flags 0 level 4 prefix `Firewall: *SYNFLOOD Blocked* '
    3 DROP all -- 0.0.0.0/0 0.0.0.0/0
     
    #13 ramindia, Jan 2, 2012
(You must log in or sign up to post here.)
Loading...
Similar Threads - Server Going down
  1. damian-d

    Server keeps going down, don't know why

    damian-d, Jan 9, 2017, in forum: General Discussion
    Replies:
    3
    Views:
    327
    cPanelMichael
    Jan 9, 2017
  2. ridgedale

    Split Domain Add Another Outgoing Mail Server

    ridgedale, Feb 19, 2017, in forum: E-mail Discussions
    Replies:
    2
    Views:
    215
    ridgedale
    Feb 22, 2017
  3. TheDjinn

    Emails still going to old email server.

    TheDjinn, Jan 29, 2017, in forum: E-mail Discussions
    Replies:
    2
    Views:
    397
    cPanelMichael
    Jan 30, 2017
  4. dallanlee

    Setting Incoming & Outgoing Mail Server For All Accounts

    dallanlee, Nov 24, 2016, in forum: E-mail Discussions
    Replies:
    1
    Views:
    433
    cPanelMichael
    Nov 28, 2016
  5. Li Cheng Chang

    Copy outgoing email to another server

    Li Cheng Chang, Nov 20, 2016, in forum: E-mail Discussions
    Replies:
    1
    Views:
    319
    cPanelMichael
    Nov 23, 2016

Share This Page