The Community Forums

Interact with an entire community of cPanel & WHM users!
  1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Strange Apache Problem

Discussion in 'EasyApache' started by hp_m00n, Mar 27, 2005.

  1. hp_m00n

    hp_m00n Member

    Joined:
    Feb 20, 2005
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    Hi!

    A reboot resolved my problem. I'm not sure if only temporarily. But I'm curious about what could have happened. Any clue?

    Here's the situation:

    I had 170 +- accounts on a Celeron 1.7 with 1 GB RAM. I got a Pentium 4 3.0 1 GB RAM and asked a security company to harden it (snort, mod_security, etc) and do the transfer.

    The hostname and nameservers on the old and new server are the same. After transfering all accounts, the ip of the nameservers was changed. Just found that they didn't change the ip of the hostname. Just changed it.

    When the ip were changed and 4 hours later, I got a first complaint about problems acessing php and mysql files. The ips were changed 15:00 hours. First complaint at 19:00 hours. The next morning I found out I had problems acessing all php and mysql sites. ANd no problem acessing non php and mysql sites.

    Example: SAME DOMAIN

    ping to the new server - dns report without issues

    www.example.com is a php and mysql forum - the browser just loaded and loaded and loaded and loaded

    www.example.com/name/ is just a listing of folders and files. Quick access

    www.example.com/1 does not exist - quick acess - url not found on this server

    This would happen at higher server load hours and at very low server load hours with not many apache request. High server load here is under 1.0.

    A apache restart just solved the problem for a few minutes or hours. The php and mysql sites would load very fast. But for 30 min / 1 hour. I didn't time it. Then they would start to load and load and load again.

    All these sites on a weaker server, with little apache optimization, worked just fine.

    Some DATA:

    When the sites start loading, loading and loading:

    ps ax | grep httpd | grep -v grep | wc -l

    53

    tail -20 /usr/local/apache/logs/error_log

    [Sat Mar 26 23:21:56 2005] [notice] child pid 1883 exit signal Segmentation fault (11)
    [Sat Mar 26 23:22:07 2005] [error] [client 68.142.251.145] File does not exist: /home/xxxxx1/public_html/demo/robots.txt
    [Sat Mar 26 23:22:07 2005] [error] [client 68.142.251.145] File does not exist: /home/xxxxx1/public_html/demo/404.shtml
    [Sat Mar 26 23:22:16 2005] [error] [client 68.142.251.73] File does not exist: /home/xxxxxx2/public_html/robots.txt
    [Sat Mar 26 23:22:16 2005] [error] [client 68.142.251.73] File does not exist: /home/xxxxxx2/public_html/404.shtml
    [Sat Mar 26 23:22:35 2005] [error] [client 213.58.103.141] File does not exist: /usr/local/apache/htdocs/favicon.ico
    [Sat Mar 26 23:22:35 2005] [error] [client 213.58.103.141] File does not exist: /usr/local/apache/htdocs/404.shtml
    [Sat Mar 26 23:23:09 2005] [error] [client 68.142.251.149] File does not exist: /home/xxx3/public_html/robots.txt
    [Sat Mar 26 23:23:09 2005] [error] [client 68.142.251.149] File does not exist: /home/xxx3/public_html/404.shtml
    [Sat Mar 26 23:24:55 2005] [error] [client 82.155.10.160] File does not exist: /home/xxxx4/public_html/favicon.ico
    [Sat Mar 26 23:24:55 2005] [error] [client 82.155.10.160] File does not exist: /home/xxxx4/public_html/404.shtml
    [Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.58] File does not exist: /home/xx5/public_html/robots.txt
    [Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.58] File does not exist: /home/xx5/public_html/404.shtml
    [Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.66] File does not exist: /home/xx5/public_html/paginas/pag_servimos.html
    [Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.66] File does not exist: /home/xx5/public_html/404.shtml
    [Sat Mar 26 23:26:31 2005] [error] [client 207.248.240.119] File does not exist: /home/xxxx6/public_html/oadmiradorsecreto/blog/archives/2004/12/segunda_dose.html
    [Sat Mar 26 23:26:31 2005] [error] [client 207.248.240.119] File does not exist: /home/xxxx6/public_html/oadmiradorsecreto/404.shtml

    Apache status from WHM

    _WWWW_WW_WKW_WWW_WWW_W__W_WWWWWWWWWWWWWWWWWWWWWW_WWWWWWW_W

    Then from WHM I restarted apache. Sites started loading fine. THis doesn't affect non php and mysql sites Those continue to load just fine.

    After apache restart

    ps ax | grep httpd | grep -v grep | wc -l

    17

    apache status from whm

    4 requests:

    ___W_K___________KK.


    Apache was recompiled by the server security company with no success. A reboot abou 10 hours might have done the job. Any clue to what happened? Thank you.
     
    #1 hp_m00n, Mar 27, 2005
    Last edited: Mar 27, 2005
  2. hp_m00n

    hp_m00n Member

    Joined:
    Feb 20, 2005
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    Problem back

    The problem is back again. Sites start to load and load and load. Restarted apache. That will give me 1 hour of normality. If I do a reboot I guess that will give more time. But then the problem returns.
     
    #2 hp_m00n, Mar 27, 2005
    Last edited: Mar 27, 2005
  3. hp_m00n

    hp_m00n Member

    Joined:
    Feb 20, 2005
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    Ram

    I've read this:

    "Signal 11 is usually* indicative of a hardware error. Most of the time,
    people who get it can make it go away by reseating the RAM in its slots,
    reducing the memory and system bus speed(s), not overclocking, etc.

    http://www.ayukov.com/essays/signal11.html
    http://www.bitwizard.nl/sig11/

    I suggest you check your hardware first, as I don't believe there have
    been any reports of invalid memory accesses in Apache. (The 1.3.x code
    tree has been around for quite a while, after all!)"

    I requested our datacenter to check RAM. They were very helpful. Changed RAM. I'll have to wait and see. Everything is ok.

    From what I read segmentantion fault 11 is related to memory access. I've read it could be bad coding or hardware, especially RAM. So if you ever get this kind of error, ask your datacenter to check RAM.
     
  4. eth00

    eth00 Well-Known Member
    PartnerNOC

    Joined:
    Mar 30, 2003
    Messages:
    723
    Likes Received:
    1
    Trophy Points:
    18
    Location:
    NC
    cPanel Access Level:
    Root Administrator
    Just as a note you can always run memtester yourself to check if the RAM is bad. Sounds like your datacenter was pretty quick to check it out but many are not...
     
  5. mr.wonderful

    mr.wonderful BANNED

    Joined:
    Feb 1, 2004
    Messages:
    345
    Likes Received:
    0
    Trophy Points:
    0
    Why dont you ask them what they did? I mean if you paid good money the least they can do is fix the issues! Why post here? Nobody knows what they installed or modified. If i was you, i would contact whoever made the changes.
     
  6. hp_m00n

    hp_m00n Member

    Joined:
    Feb 20, 2005
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    Not RAM

    Another segmentation fault at 12:30:48 machine time and another apache restart.

    I have nothing to say about the datacenter. Very quick to change RAM. But I guess it wasn't the RAM.

    Any clue? When this segmentation occured, server load was 0,06. more or less 12 hours after last reboot. But I didn't do a reboot this time. And a reboot buys me more time than an apache restart.

    I'm going to contact the company that did the security and server migration.
     
    #6 hp_m00n, Mar 28, 2005
    Last edited: Mar 28, 2005
  7. hp_m00n

    hp_m00n Member

    Joined:
    Feb 20, 2005
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    apache

    The company that did the security and setup recompiled apache from scratch. The problem remains. I sent an email asking for their position relating this matter. No response.

    I did a strace -f /usr/local/apache/bin/httpd startssl when apache hang. I'm pasting the last part. Can anyone give me a clue to where to look.

    socket(PF_FILE, SOCK_STREAM, 0) = 3
    connect(3, {sa_family=AF_FILE, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory)
    close(3) = 0
    open("/etc/hosts", O_RDONLY) = 3
    fcntl64(3, F_GETFD) = 0
    fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
    fstat64(3, {st_mode=S_IFREG|0644, st_size=191, ...}) = 0
    mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xbf7fe000
    read(3, "# Do not remove the following li"..., 4096) = 191
    read(3, "", 4096) = 0
    close(3) = 0
    munmap(0xbf7fe000, 4096) = 0
    time(NULL) = 1112030272
    write(2, "[Mon Mar 28 18:17:52 2005] [warn"..., 85) = 85
    write(2, "\n", 1) = 1
    socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
    fcntl64(3, F_DUPFD, 15) = 16
    close(3) = 0
    fcntl64(16, F_SETFD, FD_CLOEXEC) = 0
    setsockopt(16, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
    setsockopt(16, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
    setsockopt(16, SOL_TCP, TCP_NODELAY, [1], 4) = 0
    bind(16, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)
    time(NULL) = 1112030272
    write(2, "[Mon Mar 28 18:17:52 2005] [crit"..., 98) = 98
    write(2, "\n", 1) = 1
    close(16) = 0
    exit_group(1) = ?
    Process 7666 detached
     
  8. hp_m00n

    hp_m00n Member

    Joined:
    Feb 20, 2005
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    apache

    When it hangs, I can access any site html. The problem is php sites.
     
    #8 hp_m00n, Mar 28, 2005
    Last edited: Mar 28, 2005
  9. AndyReed

    AndyReed Well-Known Member
    PartnerNOC

    Joined:
    May 29, 2004
    Messages:
    2,222
    Likes Received:
    3
    Trophy Points:
    38
    Location:
    Minneapolis, MN
    It seems that you have serious problem with your server. It is rather difficult to say what and/or where the problem is within the context of all the changes made to your server.
     
    #9 AndyReed, Mar 28, 2005
    Last edited: Aug 5, 2008
  10. hp_m00n

    hp_m00n Member

    Joined:
    Feb 20, 2005
    Messages:
    8
    Likes Received:
    0
    Trophy Points:
    1
    Update

    Just an update.

    rack911.com WAS NOT the company that did the initial security configuration!!!

    But when you have a problem, who do you call? rack911.com!!!!!!

    Steven disabled the software installed by the other company, and no more Segmentation faults.

    Thank you, rack911.com!

    The other compay has a good reputation. But they messed 2 of my servers. On the other they installed mrtg, on /etc/mrtg that started to dump cores into /usr/local/apache/htpdocs/mrtg. And that program was doing a grep at my exim main log that was just messing up my load. So I shut down mrtg.

    I suppose they have done a good job elsewhere. Otherwise they wouldn't have such a good reputation. I guess sometimes bad things happen. I want ask for a refund. I'll just forget it.
     
  11. StevenC

    StevenC Well-Known Member

    Joined:
    Jan 1, 2004
    Messages:
    254
    Likes Received:
    0
    Trophy Points:
    16
    Never a problem :) Trobleshooting is what we do best :) Libsafe was the culprit.
     
  12. rvbalraj

    rvbalraj Member

    Joined:
    Aug 28, 2006
    Messages:
    19
    Likes Received:
    0
    Trophy Points:
    1
    Hi, We are also facing the same issue. and we have been trying to resolve this issue since one month.

    Can you please let me know is this problem is solved on your server. If yes, please let me know, how did you resolve it.

    Thanks,
    Raj.
     
  13. cPanelDavidG

    cPanelDavidG Technical Product Specialist

    Joined:
    Nov 29, 2006
    Messages:
    11,279
    Likes Received:
    8
    Trophy Points:
    38
    Location:
    Houston, TX
    cPanel Access Level:
    Root Administrator
    This thread is several years old so you may be better off posting a new thread or letting our technical analysts take a look at your server for you as any advice presented previously in this thread may be antiquated.
     
Loading...

Share This Page