The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Help Desperatly! Server crash every day httpd consumes all memory

Discussion in 'General Discussion' started by kran, Dec 13, 2004.

  1. kran

    kran Well-Known Member

    Joined:
    Jul 5, 2003
    Messages:
    74
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Colombia
    cPanel Access Level:
    Root Administrator
    I´ve been experiencing server lockups almost every day, when it comes back i have to restart DNS/Name (server BIND) in order to have my sites back on line. Have checked the logs and there are no indications of what´s making it crash.

    I´ve have sttoped Spamd, Analog and calmv still not help, also checked user cron jobs etc.

    Server is a RHEL 3 2.4.21-4.0.1.EL (Did the IOwait tweak)

    I Have afp, PRM and SIM installed and still no help, load goes over 28 and the server crashes, I noticed all my memory gets consumed (physical and Swap), usually i see 2 nobody, httpd processes that take all the memory before it crashes.

    I´ve to keep monitoring server loads and reset exim, httpd and mysqld when CPU goes over 10 so it will bring load down and the server will work Ok for 6, 8 or more hours, before load goes hig.


    Here the last top when it crashed.

    ***********************************************

    06:38:25 up 9:11, 1 user, load average: 28.36, 10.87, 4.44
    99 processes: 76 sleeping, 22 running, 0 zombie, 1 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 0.0% 0.0% 94.2% 0.8% 0.1% 2.9% 1.8%
    Mem: 1022480k av, 1014268k used, 8212k free, 0k shrd, 7332k buff
    967604k active, 13620k inactive
    Swap: 2097136k av, 2097136k used, 0k free 15992k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    15327 nobody 16 0 1458M 490M 504 D 1.9 49.1 0:24 0 /usr/local/apache/bin/httpd
    15325 nobody 16 0 1391M 433M 460 D 2.2 43.4 0:23 0 /usr/local/apache/bin/httpd
    16816 nobody 15 0 1784 1388 988 S 0.0 0.1 0:00 0 proftpd: connected: 12.96.160.116 (12.96.160.116:14473)
    16804 nobody 15 0 1752 1268 956 S 0.0 0.1 0:00 0 proftpd: connected: 12.96.160.116 (12.96.160.116:13071)
    10180 named 25 0 5100 1232 712 S 1.0 0.1 0:08 0 /usr/sbin/named -u named
    15350 root 35 19 8872 1204 416 D N 0.9 0.1 0:12 0 pkgacct - pereiran
    16814 nobody 16 0 14840 1076 816 R 2.5 0.1 0:00 0 /usr/local/apache/bin/httpd
    15328 nobody 15 0 17092 1036 572 D 1.0 0.1 0:00 0 /usr/local/apache/bin/httpd
    14631 nobody 16 0 18044 1000 692 R 1.0 0.0 0:00 0 /usr/local/apache/bin/httpd
    15313 nobody 16 0 15832 896 552 D 2.0 0.0 0:00 0 /usr/local/apache/bin/httpd
    16811 nobody 16 0 14700 884 680 R 2.9 0.0 0:00 0 /usr/local/apache/bin/httpd
    15214 nobody 16 0 17428 880 376 R 1.1 0.0 0:00 0 /usr/local/apache/bin/httpd
    16809 nobody 16 0 14688 844 668 R 2.4 0.0 0:00 0 /usr/local/apache/bin/httpd
    16813 nobody 16 0 14672 844 652 R 2.7 0.0 0:00 0 /usr/local/apache/bin/httpd
    5649 mailman 15 0 3812 836 416 S 0.3 0.0 0:00 0 /usr/bin/python2.2 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=BounceRunner:0:1
    15980 root 16 0 836 836 452 R 0.4 0.0 0:02 0 top
    15255 nobody 16 0 14844 836 660 R 1.5 0.0 0:00 0 /usr/local/apache/bin/httpd
    16808 nobody 16 0 14672 828 652 R 3.0 0.0 0:00 0 /usr/local/apache/bin/httpd
    16810 nobody 16 0 14620 788 600 R 2.3 0.0 0:00 0 /usr/local/apache/bin/httpd
    16826 mailnull 19 0 1008 788 648 R 2.9 0.0 0:00 0 /usr/sbin/exim -bd -q60m
    14712 nobody 16 0 15884 784 368 R 2.0 0.0 0:00 0 /usr/local/apache/bin/httpd
    16807 root 16 0 3800 772 584 R 1.7 0.0 0:00 0 webmaild - serving 64.76 restart
    16812 nobody 16 0 14560 724 540 R 3.6 0.0 0:00 0 /usr/local/apache/bin/httpd
    16817 nobody 15 0 14512 704 492 R 0.6 0.0 0:00 0 /usr/local/apache/bin/httpd
    16819 nobody 15 0 14512 704 492 R 0.9 0.0 0:00 0 /usr/local/apache/bin/httpd
    16818 nobody 16 0 14512 700 492 R 0.8 0.0 0:00 0 /usr/local/apache/bin/httpd
    16823 nobody 16 0 14548 692 528 R 3.1 0.0 0:00 0 /usr/local/apache/bin/httpd
    16822 nobody 15 0 14516 688 496 S 0.0 0.0 0:00 0 /usr/local/apache/bin/httpd
    5657 mailman 15 0 3136 684 264 S 0.0 0.0 0:00 0 /usr/bin/python2.2 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=OutgoingRunner:0
    16820 nobody 15 0 14508 680 488 S 0.0 0.0 0:00 0 /usr/local/apache/bin/httpd
    16821 nobody 15 0 14508 680 488 S 0.0 0.0 0:00 0 /usr/local/apache/bin/httpd
    15258 nobody 16 0 15760 656 372 D 2.0 0.0 0:00 0 /usr/local/apache/bin/httpd
    15312 nobody 15 0 15116 656 500 D 0.6 0.0 0:00 0 /usr/local/apache/bin/httpd
    5651 mailman 15 0 3040 648 248 D 0.2 0.0 0:00

    **************************************
    And this was yesterday
    **************************************

    19:00:10 up 3 days, 23:16, 1 user, load average: 28.62, 9.49, 3.62
    110 processes: 108 sleeping, 2 running, 0 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 0.4% 0.0% 5.7% 0.8% 0.0% 92.9% 0.0%
    Mem: 1022480k av, 1014136k used, 8344k free, 0k shrd, 5308k buff
    980736k active, 2460k inactive
    Swap: 2097136k av, 2097132k used, 4k free 13308k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    6546 nobody 15 0 1545M 478M 644 D 0.8 47.9 0:24 0 /usr/local/apache/bin/httpd -DSSL
    6553 nobody 15 0 15112 1028 796 D 0.8 0.1 0:00 0 /usr/local/apache/bin/httpd -DSSL
    6 root 15 0 0 0 0 SW 0.4 0.0 0:30 0 kscand
    32204 named 25 0 7136 1120 848 S 0.4 0.1 0:53 0 /usr/sbin/named -u named
    6539 nobody 15 0 14928 1020 784 D 0.4 0.0 0:00 0 /usr/local/apache/bin/httpd -DSSL
    6543 nobody 15 0 14928 1024 784 D 0.4 0.1 0:00 0 /usr/local/apache/bin/httpd -DSSL
    6547 nobody 15 0 1335M 452M 644 D 0.4 45.2 0:21 0 /usr/local/apache/bin/httpd -DSSL
    6589 nobody 15 0 14988 1244 860 D 0.4 0.1 0:00 0 /usr/local/apache/bin/httpd -DSSL
    6593 nobody 15 0 14896 936 768 D 0.4 0.0 0:00 0 /usr/local/apache/bin/httpd -DSSL
    6596 nobody 15 0 14896 936 768 D 0.4 0.0 0:00 0 /usr/local/apache/bin/httpd -DSSL
    6601 root 20 0 456 440 376 D 0.4 0.0 0:00 0 CROND
    1 root 15 0 112 80 60 S 0.0 0.0 0:08 0 init
    2 root 15 0 0 0 0 SW 0.0 0.0 0:02 0 keventd
    3 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kapmd
    4 root 34 19 0 0 0 SWN 0.0 0.0 0:00 0 ksoftirqd/0
    7 root 15 0 0 0 0 SW 0.0 0.0 0:01 0 bdflush
    5 root 15 0 0 0 0 DW 0.0 0.0 1:09 0 kswapd
    8 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kupdated
    9 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 mdrecoveryd
    13 root 15 0 0 0 0 SW 0.0 0.0 2:17 0 kjournald
    69 root 25 0 0 0 0 SW 0.0 0.0 0:00 0 khubd
    2940 root 15 0 0 0 0 SW 0.0 0.0 0:00 0 kjournald
    2941 root 15 0 0 0 0 SW 0.0 0.0 0:44 0 kjournald
    3628 root 15 0 384 352 304 S 0.0 0.0 0:26 0 syslogd -m 0
    3632 root 15 0 188 136 132 S 0.0 0.0 0:00 0 klogd -x
    4876 root 15 0 1776 732 488 D 0.0 0.0 0:02 0 chkservd
    4969 root 15 0 412 384 328 D 0.0 0.0 0:00 0 crond
    5275 root 34 19 11020 612 444 D N 0.0 0.0 13:22 0 cpanellogd - sleeping for logs
    5330 cpanel 15 0 1052 268 264 S 0.0 0.0 0:01 0 /usr/bin/stunnel-4.04local /usr/local/cpanel/etc/stunnel/default/stunnel.conf
    5357 mailman 15 0 3044 148 144 S 0.0 0.0 0:00 0 /usr/bin/python2.2 /usr/local/cpanel/3rdparty/mailman/bin/mailmanctl -s start
    5365 root 15 0 140 88 84 S 0.0 0.0 0:00 0 rhnsd --interval 240
    5369 mailman 15 0 4060 864 400 S 0.0 0.0 0:01 0 /usr/bin/python2.2 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=ArchRunner:0:1 -
    5370 mailman 15 0 6740 980 496 S 0.0 0.0 0:04 0 /usr/bin/python2.2 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=BounceRunner:0:1
    5371 mailman 15 0 3040 604 248 S 0.0 0.0 0:00 0 /usr/bin/python2.2 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=CommandRunner:0:
    5372 mailman 15 0 5508 908 492 D 0.0 0.0 0:04 0 /usr/bin/python2.2 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=IncomingRunner:0
    5373 mailman 15 0 3076 648 248 S 0.0 0.0 0:00 0 /usr/bin/python2.2 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=NewsRunner:0:1 -
    5374 mailman 15 0 4644 1016 548 S 0.0 0.0 0:19 0 /usr/bin/python2.2 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=OutgoingRunner:0
    5375 mailman 15 0 4424 856 452 D 0.0 0.0 0:05 0 /usr/bin/python2.2 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=VirginRunner:0:1
    5376 mailman 15 0 3040 196 192 S 0.0 0.0 0:00 0 /usr/bin/python2.2 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=RetryRunner:0:1

    ************************************

    This is driving me crazy, please any one can help me with some advice..
     
  2. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    I would do the following:

    1. Upgrade the kernel. The one you are running is slow, has bugs and has security vulnerabilities

    2. Make sure you are an up to date compilation of httpd and php (i.e. rebuild it through WHM).

    3. Check in WHM > Apache Status to see what the children hogging resources are doing

    4. When the server is under load, run the following to see if you're getting DOSed from a single IP address:

    netstat -autpn | grep :80

    If you are, install an iptables firewall (e.g. APF wth anti-dos enabled and BFD installed) and block it. This would be a good idea anyway.

    5. Install mod_dosevasive and mod_security
     
  3. kran

    kran Well-Known Member

    Joined:
    Jul 5, 2003
    Messages:
    74
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Colombia
    cPanel Access Level:
    Root Administrator
    I will follow your Advice

    I have AFP firewall, I will recompile Apache and upgrade the kernel tonight.

    Any method where i can limit max memory used by a single http process?
     
  4. NetX

    NetX Well-Known Member

    Joined:
    Jun 18, 2003
    Messages:
    177
    Likes Received:
    0
    Trophy Points:
    16
    Do all the chipry's recommendations.

    When I had this problem (I had up to 100 in load average) I fixed it loocking in Apache status what domain caused the high load and simply I cancelled the account :p .

    Sometimes there is not option, specially if your server capacity is low.

    Upgrading yur kernel will help, but my recommendation is terminate the offending account.

    Check too, the mailque, when somebody is sending spam the load is high.
     
  5. chirpy

    chirpy Well-Known Member

    Joined:
    Jun 15, 2002
    Messages:
    13,475
    Likes Received:
    20
    Trophy Points:
    38
    Location:
    Go on, have a guess
    Yes. The simplest way is to do it in WHM > Security > Modify Apache Memory Usage

    This will add a couple of lines at the top of httpd.conf which restricts child CPU and memory usage to sensible levels.
     
  6. kran

    kran Well-Known Member

    Joined:
    Jul 5, 2003
    Messages:
    74
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Colombia
    cPanel Access Level:
    Root Administrator
    Will this has anything to do?

    In var/log/messages this is a message i find when the server crashes.

    ******************************
    Dec 13 06:36:17 usa kernel: ** IN_UDP DROP ** IN=eth0 OUT= MAC=00:30:48:42:78:cc:00:d0:02:49:64:00:08:00 SRC=82.159.143.5 DST=69.93.79.50

    LEN=38 TOS=0x00 PREC
    =0x00 TTL=47 ID=1046 PROTO=UDP SPT=48308 DPT=49153 LEN=18
    Dec 13 09:10:47 usa syslogd 1.4.1: restart.
    ******************************

    also notice in the APACHE LOGS a windows attack 15 ~ 30 minutes before server becomes unresponsive

    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /scripts/root.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /MSADC/root.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /c/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /d/winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /scripts/..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /_vti_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /_mem_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET

    /msadc/..%255c../..%255c../..%255c/..%c1%1c../..%c1%1c../..%c1%1c../winnt/system32/cmd.exe?/c+dir HTTP/1.
    0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /scripts/..%c1%1c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /scripts/..%c0%2f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /scripts/..%c0%af../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /scripts/..%c1%9c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /scripts/..%%35%63../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 400 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /scripts/..%%35c../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 400 -
    69.153.229.124 - - [13/Dec/2004:06:07:38 -0500] "GET /scripts/..%25%35%63../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 -
    69.153.229.124 - - [13/Dec/2004:06:07:39 -0500] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 -
    127.0.0.1 - - [13/Dec/2004:06:10:00 -0500] "GET /whm-server-status HTTP/1.0" 200 13870
    127.0.0.1 - - [13/Dec/2004:06:14:35 -0500] "GET / HTTP/1.0" 200 2973
    127.0.0.1 - - [13/Dec/2004:06:15:01 -0500] "GET /whm-server-status HTTP/1.0" 200 13993
    127.0.0.1 - - [13/Dec/2004:06:20:00 -0500] "GET /whm-server-status HTTP/1.0" 200 13808
    127.0.0.1 - - [13/Dec/2004:06:22:56 -0500] "GET / HTTP/1.0" 200 2973
    127.0.0.1 - - [13/Dec/2004:06:25:01 -0500] "GET /whm-server-status HTTP/1.0" 200 14097
    127.0.0.1 - - [13/Dec/2004:06:30:00 -0500] "GET /whm-server-status HTTP/1.0" 200 13963
    127.0.0.1 - - [13/Dec/2004:06:31:17 -0500] "GET / HTTP/1.0" 200 2973
    127.0.0.1 - - [13/Dec/2004:06:35:00 -0500] "GET /whm-server-status HTTP/1.0" 200 14120 <------- SERVER DIED
    127.0.0.1 - - [13/Dec/2004:09:10:06 -0500] "GET /whm-server-status HTTP/1.0" 200 8019 <--------- server back on line

    ***************************
     
  7. sawbuck

    sawbuck Well-Known Member

    Joined:
    Jan 18, 2004
    Messages:
    1,367
    Likes Received:
    5
    Trophy Points:
    38
    cPanel Access Level:
    Root Administrator
    Would suggest you investigate running mod_security. Can be installed from within WHM (under Addon Modules) or you can download it from here: http://modsecurity.org/index.php
     
  8. kran

    kran Well-Known Member

    Joined:
    Jul 5, 2003
    Messages:
    74
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Colombia
    cPanel Access Level:
    Root Administrator
    Just to to let you know

    1) WHM > Security > Modify Apache Memory Usage

    2) I upgraded the kernel

    3) Installed Apache mod_security

    4) I had AFP installed (before)

    So far is running Ok, I hope this solves the problem.
     
  9. kran

    kran Well-Known Member

    Joined:
    Jul 5, 2003
    Messages:
    74
    Likes Received:
    0
    Trophy Points:
    6
    Location:
    Colombia
    cPanel Access Level:
    Root Administrator
    worked fine for 21 one days

    It started acting again has crashed 2 times, happens so fast it will not letme investigate what http process are eating up all memory.

    13:44:50 up 21 days, 12:44, 2 users, load average: 54.32, 20.21, 8.19
    135 processes: 67 sleeping, 68 running, 0 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 0.7% 0.0% 47.6% 0.7% 0.4% 50.4% 0.0%
    Mem: 1019408k av, 1011196k used, 8212k free, 0k shrd, 8436k buff
    964488k active, 14088k inactive
    Swap: 2097136k av, 2097136k used, 0k free 9636k cached

    PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
    1126 nobody 15 0 1022M 322M 480 R 0.7 32.3 0:17 0 /usr/local/apache/bin/httpd
    1037 nobody 15 0 927M 318M 480 R 0.4 31.9 0:15 0 /usr/local/apache/bin/httpd
    1065 nobody 15 0 838M 276M 480 R 0.7 27.8 0:14 0 /usr/local/apache/bin/httpd
    1180 root 15 0 3988 1428 876 R 0.4 0.1 0:00 0 cppop - serving 200.93.172.57 - AUTHORIZATION
    1166 root 15 0 4044 1376 768 R 0.7 0.1 0:00 0 cppop - serving 69.79.39.24 - AUTHORIZATION
    1157 root 15 0 4032 1372 756 R 0.0 0.1 0:00 0 cppop - serving 69.79.39.24 - AUTHORIZATION
    1165 root 15 0 4032 1360 756 R 0.2 0.1 0:00 0 cppop - serving 200.75.67.237 - AUTHORIZATION
    1167 root 15 0 4032 1356 756 R 0.2 0.1 0:00 0 cppop - serving 200.75.67.237 - AUTHORIZATION
    1168 root 15 0 4032 1356 756 R 1.1 0.1 0:00 0 cppop - serving 200.119.5.153 - AUTHORIZATION
    1188 root 15 0 3944 1356 848 R 0.2 0.1 0:00 0 cppop - serving 200.119.35.207 - GREETING
    1187 root 15 0 3924 1336 828 R 0.7 0.1 0:00 0 cppop - serving 200.24.110.32 - GREETING
    1185 root 15 0 3916 1328 820 R 0.2 0.1 0:00 0 cppop - serving 65.167.50.17 - GREETING
    1182 root 15 0 3912 1316 816 R 0.0 0.1 0:00 0 cppop - serving 200.116.96.102 - GREETING
    1181 root 15 0 3908 1304 812 R 0.7 0.1 0:00 0 cppop - serving 200.122.206.41 - GREETING
    1193 root 15 0 3916 1304 820 R 0.7 0.1 0:00 0 cppop - accepting on port 110
    1192 root 15 0 3916 1292 820 R 0.7 0.1 0:00 0 cppop - accepting on port 110
    1186 mailnull 15 0 1440 1224 1040 R 0.0 0.1 0:00 0 /usr/sbin/exim -bd -q60m
    1164 root 15 0 3960 1216 684 R 0.2 0.1 0:00 0 cppop - serving 205.160.35.80 - AUTHORIZATION
    1191 mailnull 15 0 1412 1200 1016 R 0.9 0.1 0:00 0 /usr/sbin/exim -bd -q60m
    1156 mailnull 15 0 1464 1176 988 R 0.2 0.1 0:00 0 /usr/sbin/exim -bd -q60m
    1150 mailnull 15 0 1444 1160 980 R 0.2 0.1 0:00 0 /usr/sbin/exim -bd -q60m
    1170 root 15 0 3840 1156 732 R 1.1 0.1 0:00 0 cppop - serving 200.93.150.37 - AUTHORIZATION
    1178 root 15 0 3800 1100 704 R 0.9 0.1 0:00 0 cppop - accepting on port 110
    1155 root 15 0 3796 1096 696 R 0.4 0.1 0:00 0 cppop - serving 200.119.5.153 - AUTHORIZATION
    1159 root 15 0 3800 1092 696 R 0.4 0.1 0:00 0 cppop - serving 200.93.172.57 - AUTHORIZATION
    1158 root 15 0 3784 1068 680 R 0.0 0.1 0:00 0 cppop - serving 200.119.5.153 - AUTHORIZATION
     
  10. simone70

    simone70 Member

    Joined:
    Nov 3, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    same problems

    I'm experiencing similar problems.
    any news ?

    cannot neither login in ssh, I've seen that processes number increase quickly and swap reach 100%.

    The only solution rebooting the server.

    I'm running fedora 2.4.26 and got many gfp=0x1d2/0 gfp=0xf0/0 errors in /var/log/messages.
     
  11. robjs

    robjs Registered
    PartnerNOC

    Joined:
    Oct 16, 2004
    Messages:
    3
    Likes Received:
    0
    Trophy Points:
    1
    Can you guys try disabling mod_rewrite on your server for a bit? Because, I've got exactly the same problem, I've nailed it down to problems with mod_rewrite, and I'm running RHEL too.

    Steps to diagnose:

    1. Enable hotlink protection on a site
    2. Try and load a file from that site.
    3. If the apache thread hangs, and you never get the file then you're likely to have the same problem as me. You'll notice in top that there's an apache thread using around 25% CPU.

    So;

    1. Comment out mod_rewrite from your httpd.conf
    2. Turn off hotlink protection on the site.
    3. Restart apache.
    4. Try and reload the site, using the URL you used before. If the file loads then I'd say it's definitely the same problem.

    You can verify by;

    1. Uncommenting mod_rewrite again in the httpd.conf
    2. Turn on hotlink protection.
    3. Restart apache
    4. Change RewriteRule .*\.(jpg|jpeg|gif|png|bmp)$ - [F,NC] to RewriteRule .*\.(bmp|jpg|jpeg|gif|png|test)$ - [F,NC] (leaving the URL that'll be there in).
    5. Try and load the file, if it doesn't load, bingo, come back here screaming you have the same issue as me.

    Rob
     
  12. simone70

    simone70 Member

    Joined:
    Nov 3, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    no mod_rewrite here. :(

    I'm monitoring my Swap/Mem relation with mrtg and I've noticed that swap keep increasing costantly without releasing space when memory load is low.

    Yesterday reached the 20% and server crashed.
    In those cases I'm not able neither to login in ssh.
    Ping works but no services respond, all time out.


    this is a top of a few weeks ago:

    10:09:52 up 1 day, 8:51, 1 user, load average: 0.66, 0.87, 0.44
    203 processes: 202 sleeping, 1 running, 0 zombie, 0 stopped
    CPU states: cpu user nice system irq softirq iowait idle
    total 1.0% 0.0% 3.9% 0.0% 0.0% 0.0% 94.9%
    Mem: 188188k av, 183444k used, 4744k free, 0k shrd, 1508k buff
    9616k active, 154856k inactive
    Swap: 262136k av, 262136k used, 0k free 6448k cached

    my usual processes nr is around 55 and memory load during the day aprox 80%.
     
  13. simone70

    simone70 Member

    Joined:
    Nov 3, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    anyone ?
    I'm still having problems... :mad:
     
  14. gordonrp

    gordonrp Member

    Joined:
    Oct 14, 2004
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    I am having the same problems. Server load goes as high as 105. I do not have phpbb installed. I have whm 10. I have just installed mod_security. I already have bfd and apf installed.

    The rkhunter shows the machine is clean..

    Any ideas?
    gp
     
  15. ayik

    ayik Member
    PartnerNOC

    Joined:
    Oct 7, 2004
    Messages:
    15
    Likes Received:
    0
    Trophy Points:
    1
    i am experiencing similar httpd problem too, httpd consumes ALL CPU. Especially when it start, restart (SIGUSR1). During this period (for about 5 minutes restart process), CPU Usage is 90%, load avg is > 4.0 (night) or > 9.0 (day) and no site can be browsed (connection refused). httpd is not sleeping, it is Running (R), i think he is parsing my httpd.conf at those time, (but it took more than 5 minutes!)

    I wonder how much user in your server kran? I am having 1014 user now, and that's what is being blamed by me. But, some days ago (before 20 jan, before i upgraded the whm), i am having 980 users and the server run well (load avg is only 0.8)

    Here the top at 6 am morning (w/ only few httpd visitors!, usually i got 190 tasks and load avg is still 0.9 at day time):
    ======================================================
    top - 06:39:15 up 8:10, 2 users, load average: 2.97, 2.44, 2.03
    Tasks: 113 total, 3 running, 108 sleeping, 2 stopped, 0 zombie
    Cpu0 : 91.4% us, 0.8% sy, 0.0% ni, 7.6% id, 0.0% wa, 0.0% hi, 0.2% si
    Cpu1 : 92.4% us, 4.0% sy, 0.0% ni, 3.6% id, 0.0% wa, 0.0% hi, 0.0% si
    Mem: 1025892k total, 959472k used, 66420k free, 119376k buffers
    Swap: 2048276k total, 56k used, 2048220k free, 472076k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    21076 root 25 0 5488 3948 1744 R 91.7 0.4 1:54.01 mkvhostspasswd
    21249 root 25 0 36656 26m 3552 R 84.1 2.6 0:12.42 httpd
    =======================================================

    additional info:
    CPU is dual P4 2.8 GHz, 1GB RAM, 2x80 GB IDE, Fedora Core 2

    kernel (is this a kernel problem? because i have 2 fedora+cpanel server, one with older kernel is running smoothly, they are in the same cpanel version)
    [root@centaur root]# uname -a
    Linux centaur.idwebhost.com 2.6.10-1.9_FC2smp #1 SMP Thu Jan 13 18:15:45 EST 2005 i686 i686 i386 GNU/Linux

    WHM 10.0.0 cPanel 10.0.0-R7
    Fedora i686 - WHM X v3.1.0

    My httpd.conf size is:
    File: `/etc/httpd/conf/httpd.conf'
    Size: 1111425

    Lines Count: [root@centaur root]# wc -l /etc/httpd/conf/httpd.conf
    38231 /etc/httpd/conf/httpd.conf
     
  16. MMarko

    MMarko Well-Known Member

    Joined:
    Apr 18, 2005
    Messages:
    316
    Likes Received:
    0
    Trophy Points:
    16
    I had this problem. Have you found some info to solve this? I need mod_rewrite.
     
Loading...

Share This Page