Strange Apache Problem

[hp_m00n]

Hi!

A reboot resolved my problem. I'm not sure if only temporarily. But I'm curious about what could have happened. Any clue?

Here's the situation:

I had 170 +- accounts on a Celeron 1.7 with 1 GB RAM. I got a Pentium 4 3.0 1 GB RAM and asked a security company to harden it (snort, mod_security, etc) and do the transfer.

The hostname and nameservers on the old and new server are the same. After transfering all accounts, the ip of the nameservers was changed. Just found that they didn't change the ip of the hostname. Just changed it.

When the ip were changed and 4 hours later, I got a first complaint about problems acessing php and mysql files. The ips were changed 15:00 hours. First complaint at 19:00 hours. The next morning I found out I had problems acessing all php and mysql sites. ANd no problem acessing non php and mysql sites.

Example: SAME DOMAIN

ping to the new server - dns report without issues

www.example.com is a php and mysql forum - the browser just loaded and loaded and loaded and loaded

www.example.com/name/ is just a listing of folders and files. Quick access

www.example.com/1 does not exist - quick acess - url not found on this server

This would happen at higher server load hours and at very low server load hours with not many apache request. High server load here is under 1.0.

A apache restart just solved the problem for a few minutes or hours. The php and mysql sites would load very fast. But for 30 min / 1 hour. I didn't time it. Then they would start to load and load and load again.

All these sites on a weaker server, with little apache optimization, worked just fine.

Some DATA:

When the sites start loading, loading and loading:

ps ax | grep httpd | grep -v grep | wc -l

53

tail -20 /usr/local/apache/logs/error_log

[Sat Mar 26 23:21:56 2005] [notice] child pid 1883 exit signal Segmentation fault (11)
[Sat Mar 26 23:22:07 2005] [error] [client 68.142.251.145] File does not exist: /home/xxxxx1/public_html/demo/robots.txt
[Sat Mar 26 23:22:07 2005] [error] [client 68.142.251.145] File does not exist: /home/xxxxx1/public_html/demo/404.shtml
[Sat Mar 26 23:22:16 2005] [error] [client 68.142.251.73] File does not exist: /home/xxxxxx2/public_html/robots.txt
[Sat Mar 26 23:22:16 2005] [error] [client 68.142.251.73] File does not exist: /home/xxxxxx2/public_html/404.shtml
[Sat Mar 26 23:22:35 2005] [error] [client 213.58.103.141] File does not exist: /usr/local/apache/htdocs/favicon.ico
[Sat Mar 26 23:22:35 2005] [error] [client 213.58.103.141] File does not exist: /usr/local/apache/htdocs/404.shtml
[Sat Mar 26 23:23:09 2005] [error] [client 68.142.251.149] File does not exist: /home/xxx3/public_html/robots.txt
[Sat Mar 26 23:23:09 2005] [error] [client 68.142.251.149] File does not exist: /home/xxx3/public_html/404.shtml
[Sat Mar 26 23:24:55 2005] [error] [client 82.155.10.160] File does not exist: /home/xxxx4/public_html/favicon.ico
[Sat Mar 26 23:24:55 2005] [error] [client 82.155.10.160] File does not exist: /home/xxxx4/public_html/404.shtml
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.58] File does not exist: /home/xx5/public_html/robots.txt
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.58] File does not exist: /home/xx5/public_html/404.shtml
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.66] File does not exist: /home/xx5/public_html/paginas/pag_servimos.html
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.66] File does not exist: /home/xx5/public_html/404.shtml
[Sat Mar 26 23:26:31 2005] [error] [client 207.248.240.119] File does not exist: /home/xxxx6/public_html/oadmiradorsecreto/blog/archives/2004/12/segunda_dose.html
[Sat Mar 26 23:26:31 2005] [error] [client 207.248.240.119] File does not exist: /home/xxxx6/public_html/oadmiradorsecreto/404.shtml

Apache status from WHM

_WWWW_WW_WKW_WWW_WWW_W__W_WWWWWWWWWWWWWWWWWWWWWW_WWWWWWW_W

Then from WHM I restarted apache. Sites started loading fine. THis doesn't affect non php and mysql sites Those continue to load just fine.

After apache restart

ps ax | grep httpd | grep -v grep | wc -l

17

apache status from whm

4 requests:

___W_K___________KK.


Apache was recompiled by the server security company with no success. A reboot abou 10 hours might have done the job. Any clue to what happened? Thank you.

 

[hp_m00n]

Problem back

The problem is back again. Sites start to load and load and load. Restarted apache. That will give me 1 hour of normality. If I do a reboot I guess that will give more time. But then the problem returns.

 

[hp_m00n]

Ram

I've read this:

"Signal 11 is usually* indicative of a hardware error. Most of the time,
people who get it can make it go away by reseating the RAM in its slots,
reducing the memory and system bus speed(s), not overclocking, etc.

http://www.ayukov.com/essays/signal11.html
http://www.bitwizard.nl/sig11/

I suggest you check your hardware first, as I don't believe there have
been any reports of invalid memory accesses in Apache. (The 1.3.x code
tree has been around for quite a while, after all!)"

I requested our datacenter to check RAM. They were very helpful. Changed RAM. I'll have to wait and see. Everything is ok.

From what I read segmentantion fault 11 is related to memory access. I've read it could be bad coding or hardware, especially RAM. So if you ever get this kind of error, ask your datacenter to check RAM.

 

[eth00]

Just as a note you can always run memtester yourself to check if the RAM is bad. Sounds like your datacenter was pretty quick to check it out but many are not...

 

[mr.wonderful]

Hi!

A reboot resolved my problem. I'm not sure if only temporarily. But I'm curious about what could have happened. Any clue?

Here's the situation:

I had 170 +- accounts on a Celeron 1.7 with 1 GB RAM. I got a Pentium 4 3.0 1 GB RAM and asked a security company to harden it (snort, mod_security, etc) and do the transfer.

The hostname and nameservers on the old and new server are the same. After transfering all accounts, the ip of the nameservers was changed. Just found that they didn't change the ip of the hostname. Just changed it.

When the ip were changed and 4 hours later, I got a first complaint about problems acessing php and mysql files. The ips were changed 15:00 hours. First complaint at 19:00 hours. The next morning I found out I had problems acessing all php and mysql sites. ANd no problem acessing non php and mysql sites.

Example: SAME DOMAIN

ping to the new server - dns report without issues

www.example.com is a php and mysql forum - the browser just loaded and loaded and loaded and loaded

www.example.com/name/ is just a listing of folders and files. Quick access

www.example.com/1 does not exist - quick acess - url not found on this server

This would happen at higher server load hours and at very low server load hours with not many apache request. High server load here is under 1.0.

A apache restart just solved the problem for a few minutes or hours. The php and mysql sites would load very fast. But for 30 min / 1 hour. I didn't time it. Then they would start to load and load and load again.

All these sites on a weaker server, with little apache optimization, worked just fine.

Some DATA:

When the sites start loading, loading and loading:

ps ax | grep httpd | grep -v grep | wc -l

53

tail -20 /usr/local/apache/logs/error_log

[Sat Mar 26 23:21:56 2005] [notice] child pid 1883 exit signal Segmentation fault (11)
[Sat Mar 26 23:22:07 2005] [error] [client 68.142.251.145] File does not exist: /home/xxxxx1/public_html/demo/robots.txt
[Sat Mar 26 23:22:07 2005] [error] [client 68.142.251.145] File does not exist: /home/xxxxx1/public_html/demo/404.shtml
[Sat Mar 26 23:22:16 2005] [error] [client 68.142.251.73] File does not exist: /home/xxxxxx2/public_html/robots.txt
[Sat Mar 26 23:22:16 2005] [error] [client 68.142.251.73] File does not exist: /home/xxxxxx2/public_html/404.shtml
[Sat Mar 26 23:22:35 2005] [error] [client 213.58.103.141] File does not exist: /usr/local/apache/htdocs/favicon.ico
[Sat Mar 26 23:22:35 2005] [error] [client 213.58.103.141] File does not exist: /usr/local/apache/htdocs/404.shtml
[Sat Mar 26 23:23:09 2005] [error] [client 68.142.251.149] File does not exist: /home/xxx3/public_html/robots.txt
[Sat Mar 26 23:23:09 2005] [error] [client 68.142.251.149] File does not exist: /home/xxx3/public_html/404.shtml
[Sat Mar 26 23:24:55 2005] [error] [client 82.155.10.160] File does not exist: /home/xxxx4/public_html/favicon.ico
[Sat Mar 26 23:24:55 2005] [error] [client 82.155.10.160] File does not exist: /home/xxxx4/public_html/404.shtml
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.58] File does not exist: /home/xx5/public_html/robots.txt
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.58] File does not exist: /home/xx5/public_html/404.shtml
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.66] File does not exist: /home/xx5/public_html/paginas/pag_servimos.html
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.66] File does not exist: /home/xx5/public_html/404.shtml
[Sat Mar 26 23:26:31 2005] [error] [client 207.248.240.119] File does not exist: /home/xxxx6/public_html/oadmiradorsecreto/blog/archives/2004/12/segunda_dose.html
[Sat Mar 26 23:26:31 2005] [error] [client 207.248.240.119] File does not exist: /home/xxxx6/public_html/oadmiradorsecreto/404.shtml

Apache status from WHM

_WWWW_WW_WKW_WWW_WWW_W__W_WWWWWWWWWWWWWWWWWWWWWW_WWWWWWW_W

Then from WHM I restarted apache. Sites started loading fine. THis doesn't affect non php and mysql sites Those continue to load just fine.

After apache restart

ps ax | grep httpd | grep -v grep | wc -l

17

apache status from whm

4 requests:

___W_K___________KK.


Apache was recompiled by the server security company with no success. A reboot abou 10 hours might have done the job. Any clue to what happened? Thank you.

Why dont you ask them what they did? I mean if you paid good money the least they can do is fix the issues! Why post here? Nobody knows what they installed or modified. If i was you, i would contact whoever made the changes.

 

[hp_m00n]

Not RAM

Another segmentation fault at 12:30:48 machine time and another apache restart.

I have nothing to say about the datacenter. Very quick to change RAM. But I guess it wasn't the RAM.

Any clue? When this segmentation occured, server load was 0,06. more or less 12 hours after last reboot. But I didn't do a reboot this time. And a reboot buys me more time than an apache restart.

I'm going to contact the company that did the security and server migration.

 

[hp_m00n]

apache

The company that did the security and setup recompiled apache from scratch. The problem remains. I sent an email asking for their position relating this matter. No response.

I did a strace -f /usr/local/apache/bin/httpd startssl when apache hang. I'm pasting the last part. Can anyone give me a clue to where to look.

socket(PF_FILE, SOCK_STREAM, 0) = 3
connect(3, {sa_family=AF_FILE, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory)
close(3) = 0
open("/etc/hosts", O_RDONLY) = 3
fcntl64(3, F_GETFD) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=191, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xbf7fe000
read(3, "# Do not remove the following li"..., 4096) = 191
read(3, "", 4096) = 0
close(3) = 0
munmap(0xbf7fe000, 4096) = 0
time(NULL) = 1112030272
write(2, "[Mon Mar 28 18:17:52 2005] [warn"..., 85) = 85
write(2, "\n", 1) = 1
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
fcntl64(3, F_DUPFD, 15) = 16
close(3) = 0
fcntl64(16, F_SETFD, FD_CLOEXEC) = 0
setsockopt(16, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(16, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(16, SOL_TCP, TCP_NODELAY, [1], 4) = 0
bind(16, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)
time(NULL) = 1112030272
write(2, "[Mon Mar 28 18:17:52 2005] [crit"..., 98) = 98
write(2, "\n", 1) = 1
close(16) = 0
exit_group(1) = ?
Process 7666 detached

 

[hp_m00n]

apache

When it hangs, I can access any site html. The problem is php sites.

 

[AndyReed]

It seems that you have serious problem with your server. It is rather difficult to say what and/or where the problem is within the context of all the changes made to your server.

 

[hp_m00n]

Update

Just an update.

rack911.com WAS NOT the company that did the initial security configuration!!!

But when you have a problem, who do you call? rack911.com!!!!!!

Steven disabled the software installed by the other company, and no more Segmentation faults.

Thank you, rack911.com!

The other compay has a good reputation. But they messed 2 of my servers. On the other they installed mrtg, on /etc/mrtg that started to dump cores into /usr/local/apache/htpdocs/mrtg. And that program was doing a grep at my exim main log that was just messing up my load. So I shut down mrtg.

I suppose they have done a good job elsewhere. Otherwise they wouldn't have such a good reputation. I guess sometimes bad things happen. I want ask for a refund. I'll just forget it.

 

[StevenC]

Never a problem Trobleshooting is what we do best Libsafe was the culprit.

 

[rvbalraj]

Just an update.

rack911.com WAS NOT the company that did the initial security configuration!!!

But when you have a problem, who do you call? rack911.com!!!!!!

Steven disabled the software installed by the other company, and no more Segmentation faults.

Thank you, rack911.com!

The other compay has a good reputation. But they messed 2 of my servers. On the other they installed mrtg, on /etc/mrtg that started to dump cores into /usr/local/apache/htpdocs/mrtg. And that program was doing a grep at my exim main log that was just messing up my load. So I shut down mrtg.

I suppose they have done a good job elsewhere. Otherwise they wouldn't have such a good reputation. I guess sometimes bad things happen. I want ask for a refund. I'll just forget it.

Hi, We are also facing the same issue. and we have been trying to resolve this issue since one month.

Can you please let me know is this problem is solved on your server. If yes, please let me know, how did you resolve it.

Thanks,
Raj.

 

[cPanelDavidG]

Hi, We are also facing the same issue. and we have been trying to resolve this issue since one month.

Can you please let me know is this problem is solved on your server. If yes, please let me know, how did you resolve it.

Thanks,
Raj.

This thread is several years old so you may be better off posting a new thread or letting our technical analysts take a look at your server for you as any advice presented previously in this thread may be antiquated.