hp_m00n

Member
Feb 20, 2005
8
0
151
Hi!

A reboot resolved my problem. I'm not sure if only temporarily. But I'm curious about what could have happened. Any clue?

Here's the situation:

I had 170 +- accounts on a Celeron 1.7 with 1 GB RAM. I got a Pentium 4 3.0 1 GB RAM and asked a security company to harden it (snort, mod_security, etc) and do the transfer.

The hostname and nameservers on the old and new server are the same. After transfering all accounts, the ip of the nameservers was changed. Just found that they didn't change the ip of the hostname. Just changed it.

When the ip were changed and 4 hours later, I got a first complaint about problems acessing php and mysql files. The ips were changed 15:00 hours. First complaint at 19:00 hours. The next morning I found out I had problems acessing all php and mysql sites. ANd no problem acessing non php and mysql sites.

Example: SAME DOMAIN

ping to the new server - dns report without issues

www.example.com is a php and mysql forum - the browser just loaded and loaded and loaded and loaded

www.example.com/name/ is just a listing of folders and files. Quick access

www.example.com/1 does not exist - quick acess - url not found on this server

This would happen at higher server load hours and at very low server load hours with not many apache request. High server load here is under 1.0.

A apache restart just solved the problem for a few minutes or hours. The php and mysql sites would load very fast. But for 30 min / 1 hour. I didn't time it. Then they would start to load and load and load again.

All these sites on a weaker server, with little apache optimization, worked just fine.

Some DATA:

When the sites start loading, loading and loading:

ps ax | grep httpd | grep -v grep | wc -l

53

tail -20 /usr/local/apache/logs/error_log

[Sat Mar 26 23:21:56 2005] [notice] child pid 1883 exit signal Segmentation fault (11)
[Sat Mar 26 23:22:07 2005] [error] [client 68.142.251.145] File does not exist: /home/xxxxx1/public_html/demo/robots.txt
[Sat Mar 26 23:22:07 2005] [error] [client 68.142.251.145] File does not exist: /home/xxxxx1/public_html/demo/404.shtml
[Sat Mar 26 23:22:16 2005] [error] [client 68.142.251.73] File does not exist: /home/xxxxxx2/public_html/robots.txt
[Sat Mar 26 23:22:16 2005] [error] [client 68.142.251.73] File does not exist: /home/xxxxxx2/public_html/404.shtml
[Sat Mar 26 23:22:35 2005] [error] [client 213.58.103.141] File does not exist: /usr/local/apache/htdocs/favicon.ico
[Sat Mar 26 23:22:35 2005] [error] [client 213.58.103.141] File does not exist: /usr/local/apache/htdocs/404.shtml
[Sat Mar 26 23:23:09 2005] [error] [client 68.142.251.149] File does not exist: /home/xxx3/public_html/robots.txt
[Sat Mar 26 23:23:09 2005] [error] [client 68.142.251.149] File does not exist: /home/xxx3/public_html/404.shtml
[Sat Mar 26 23:24:55 2005] [error] [client 82.155.10.160] File does not exist: /home/xxxx4/public_html/favicon.ico
[Sat Mar 26 23:24:55 2005] [error] [client 82.155.10.160] File does not exist: /home/xxxx4/public_html/404.shtml
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.58] File does not exist: /home/xx5/public_html/robots.txt
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.58] File does not exist: /home/xx5/public_html/404.shtml
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.66] File does not exist: /home/xx5/public_html/paginas/pag_servimos.html
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.66] File does not exist: /home/xx5/public_html/404.shtml
[Sat Mar 26 23:26:31 2005] [error] [client 207.248.240.119] File does not exist: /home/xxxx6/public_html/oadmiradorsecreto/blog/archives/2004/12/segunda_dose.html
[Sat Mar 26 23:26:31 2005] [error] [client 207.248.240.119] File does not exist: /home/xxxx6/public_html/oadmiradorsecreto/404.shtml

Apache status from WHM

_WWWW_WW_WKW_WWW_WWW_W__W_WWWWWWWWWWWWWWWWWWWWWW_WWWWWWW_W

Then from WHM I restarted apache. Sites started loading fine. THis doesn't affect non php and mysql sites Those continue to load just fine.

After apache restart

ps ax | grep httpd | grep -v grep | wc -l

17

apache status from whm

4 requests:

___W_K___________KK.


Apache was recompiled by the server security company with no success. A reboot abou 10 hours might have done the job. Any clue to what happened? Thank you.
 
Last edited:

hp_m00n

Member
Feb 20, 2005
8
0
151
Problem back

The problem is back again. Sites start to load and load and load. Restarted apache. That will give me 1 hour of normality. If I do a reboot I guess that will give more time. But then the problem returns.
 
Last edited:

hp_m00n

Member
Feb 20, 2005
8
0
151
Ram

I've read this:

"Signal 11 is usually* indicative of a hardware error. Most of the time,
people who get it can make it go away by reseating the RAM in its slots,
reducing the memory and system bus speed(s), not overclocking, etc.

http://www.ayukov.com/essays/signal11.html
http://www.bitwizard.nl/sig11/

I suggest you check your hardware first, as I don't believe there have
been any reports of invalid memory accesses in Apache. (The 1.3.x code
tree has been around for quite a while, after all!)"

I requested our datacenter to check RAM. They were very helpful. Changed RAM. I'll have to wait and see. Everything is ok.

From what I read segmentantion fault 11 is related to memory access. I've read it could be bad coding or hardware, especially RAM. So if you ever get this kind of error, ask your datacenter to check RAM.
 

eth00

Well-Known Member
PartnerNOC
Mar 30, 2003
721
1
168
NC
cPanel Access Level
Root Administrator
Just as a note you can always run memtester yourself to check if the RAM is bad. Sounds like your datacenter was pretty quick to check it out but many are not...
 

mr.wonderful

BANNED
Feb 1, 2004
344
1
166
hp_m00n said:
Hi!

A reboot resolved my problem. I'm not sure if only temporarily. But I'm curious about what could have happened. Any clue?

Here's the situation:

I had 170 +- accounts on a Celeron 1.7 with 1 GB RAM. I got a Pentium 4 3.0 1 GB RAM and asked a security company to harden it (snort, mod_security, etc) and do the transfer.

The hostname and nameservers on the old and new server are the same. After transfering all accounts, the ip of the nameservers was changed. Just found that they didn't change the ip of the hostname. Just changed it.

When the ip were changed and 4 hours later, I got a first complaint about problems acessing php and mysql files. The ips were changed 15:00 hours. First complaint at 19:00 hours. The next morning I found out I had problems acessing all php and mysql sites. ANd no problem acessing non php and mysql sites.

Example: SAME DOMAIN

ping to the new server - dns report without issues

www.example.com is a php and mysql forum - the browser just loaded and loaded and loaded and loaded

www.example.com/name/ is just a listing of folders and files. Quick access

www.example.com/1 does not exist - quick acess - url not found on this server

This would happen at higher server load hours and at very low server load hours with not many apache request. High server load here is under 1.0.

A apache restart just solved the problem for a few minutes or hours. The php and mysql sites would load very fast. But for 30 min / 1 hour. I didn't time it. Then they would start to load and load and load again.

All these sites on a weaker server, with little apache optimization, worked just fine.

Some DATA:

When the sites start loading, loading and loading:

ps ax | grep httpd | grep -v grep | wc -l

53

tail -20 /usr/local/apache/logs/error_log

[Sat Mar 26 23:21:56 2005] [notice] child pid 1883 exit signal Segmentation fault (11)
[Sat Mar 26 23:22:07 2005] [error] [client 68.142.251.145] File does not exist: /home/xxxxx1/public_html/demo/robots.txt
[Sat Mar 26 23:22:07 2005] [error] [client 68.142.251.145] File does not exist: /home/xxxxx1/public_html/demo/404.shtml
[Sat Mar 26 23:22:16 2005] [error] [client 68.142.251.73] File does not exist: /home/xxxxxx2/public_html/robots.txt
[Sat Mar 26 23:22:16 2005] [error] [client 68.142.251.73] File does not exist: /home/xxxxxx2/public_html/404.shtml
[Sat Mar 26 23:22:35 2005] [error] [client 213.58.103.141] File does not exist: /usr/local/apache/htdocs/favicon.ico
[Sat Mar 26 23:22:35 2005] [error] [client 213.58.103.141] File does not exist: /usr/local/apache/htdocs/404.shtml
[Sat Mar 26 23:23:09 2005] [error] [client 68.142.251.149] File does not exist: /home/xxx3/public_html/robots.txt
[Sat Mar 26 23:23:09 2005] [error] [client 68.142.251.149] File does not exist: /home/xxx3/public_html/404.shtml
[Sat Mar 26 23:24:55 2005] [error] [client 82.155.10.160] File does not exist: /home/xxxx4/public_html/favicon.ico
[Sat Mar 26 23:24:55 2005] [error] [client 82.155.10.160] File does not exist: /home/xxxx4/public_html/404.shtml
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.58] File does not exist: /home/xx5/public_html/robots.txt
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.58] File does not exist: /home/xx5/public_html/404.shtml
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.66] File does not exist: /home/xx5/public_html/paginas/pag_servimos.html
[Sat Mar 26 23:25:09 2005] [error] [client 68.142.250.66] File does not exist: /home/xx5/public_html/404.shtml
[Sat Mar 26 23:26:31 2005] [error] [client 207.248.240.119] File does not exist: /home/xxxx6/public_html/oadmiradorsecreto/blog/archives/2004/12/segunda_dose.html
[Sat Mar 26 23:26:31 2005] [error] [client 207.248.240.119] File does not exist: /home/xxxx6/public_html/oadmiradorsecreto/404.shtml

Apache status from WHM

_WWWW_WW_WKW_WWW_WWW_W__W_WWWWWWWWWWWWWWWWWWWWWW_WWWWWWW_W

Then from WHM I restarted apache. Sites started loading fine. THis doesn't affect non php and mysql sites Those continue to load just fine.

After apache restart

ps ax | grep httpd | grep -v grep | wc -l

17

apache status from whm

4 requests:

___W_K___________KK.


Apache was recompiled by the server security company with no success. A reboot abou 10 hours might have done the job. Any clue to what happened? Thank you.
Why dont you ask them what they did? I mean if you paid good money the least they can do is fix the issues! Why post here? Nobody knows what they installed or modified. If i was you, i would contact whoever made the changes.
 

hp_m00n

Member
Feb 20, 2005
8
0
151
Not RAM

Another segmentation fault at 12:30:48 machine time and another apache restart.

I have nothing to say about the datacenter. Very quick to change RAM. But I guess it wasn't the RAM.

Any clue? When this segmentation occured, server load was 0,06. more or less 12 hours after last reboot. But I didn't do a reboot this time. And a reboot buys me more time than an apache restart.

I'm going to contact the company that did the security and server migration.
 
Last edited:

hp_m00n

Member
Feb 20, 2005
8
0
151
apache

The company that did the security and setup recompiled apache from scratch. The problem remains. I sent an email asking for their position relating this matter. No response.

I did a strace -f /usr/local/apache/bin/httpd startssl when apache hang. I'm pasting the last part. Can anyone give me a clue to where to look.

socket(PF_FILE, SOCK_STREAM, 0) = 3
connect(3, {sa_family=AF_FILE, path="/var/run/.nscd_socket"}, 110) = -1 ENOENT (No such file or directory)
close(3) = 0
open("/etc/hosts", O_RDONLY) = 3
fcntl64(3, F_GETFD) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC) = 0
fstat64(3, {st_mode=S_IFREG|0644, st_size=191, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xbf7fe000
read(3, "# Do not remove the following li"..., 4096) = 191
read(3, "", 4096) = 0
close(3) = 0
munmap(0xbf7fe000, 4096) = 0
time(NULL) = 1112030272
write(2, "[Mon Mar 28 18:17:52 2005] [warn"..., 85) = 85
write(2, "\n", 1) = 1
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
fcntl64(3, F_DUPFD, 15) = 16
close(3) = 0
fcntl64(16, F_SETFD, FD_CLOEXEC) = 0
setsockopt(16, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(16, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
setsockopt(16, SOL_TCP, TCP_NODELAY, [1], 4) = 0
bind(16, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)
time(NULL) = 1112030272
write(2, "[Mon Mar 28 18:17:52 2005] [crit"..., 98) = 98
write(2, "\n", 1) = 1
close(16) = 0
exit_group(1) = ?
Process 7666 detached
 

hp_m00n

Member
Feb 20, 2005
8
0
151
apache

When it hangs, I can access any site html. The problem is php sites.
 
Last edited:

AndyReed

Well-Known Member
PartnerNOC
May 29, 2004
2,217
4
193
Minneapolis, MN
It seems that you have serious problem with your server. It is rather difficult to say what and/or where the problem is within the context of all the changes made to your server.
 
Last edited:

hp_m00n

Member
Feb 20, 2005
8
0
151
Update

Just an update.

rack911.com WAS NOT the company that did the initial security configuration!!!

But when you have a problem, who do you call? rack911.com!!!!!!

Steven disabled the software installed by the other company, and no more Segmentation faults.

Thank you, rack911.com!

The other compay has a good reputation. But they messed 2 of my servers. On the other they installed mrtg, on /etc/mrtg that started to dump cores into /usr/local/apache/htpdocs/mrtg. And that program was doing a grep at my exim main log that was just messing up my load. So I shut down mrtg.

I suppose they have done a good job elsewhere. Otherwise they wouldn't have such a good reputation. I guess sometimes bad things happen. I want ask for a refund. I'll just forget it.
 

StevenC

Well-Known Member
Jan 1, 2004
252
0
166
Never a problem :) Trobleshooting is what we do best :) Libsafe was the culprit.
 

rvbalraj

Member
Aug 28, 2006
19
0
151
Just an update.

rack911.com WAS NOT the company that did the initial security configuration!!!

But when you have a problem, who do you call? rack911.com!!!!!!

Steven disabled the software installed by the other company, and no more Segmentation faults.

Thank you, rack911.com!

The other compay has a good reputation. But they messed 2 of my servers. On the other they installed mrtg, on /etc/mrtg that started to dump cores into /usr/local/apache/htpdocs/mrtg. And that program was doing a grep at my exim main log that was just messing up my load. So I shut down mrtg.

I suppose they have done a good job elsewhere. Otherwise they wouldn't have such a good reputation. I guess sometimes bad things happen. I want ask for a refund. I'll just forget it.
Hi, We are also facing the same issue. and we have been trying to resolve this issue since one month.

Can you please let me know is this problem is solved on your server. If yes, please let me know, how did you resolve it.

Thanks,
Raj.
 

cPanelDavidG

Technical Product Specialist
Nov 29, 2006
11,212
13
313
Houston, TX
cPanel Access Level
Root Administrator
Hi, We are also facing the same issue. and we have been trying to resolve this issue since one month.

Can you please let me know is this problem is solved on your server. If yes, please let me know, how did you resolve it.

Thanks,
Raj.
This thread is several years old so you may be better off posting a new thread or letting our technical analysts take a look at your server for you as any advice presented previously in this thread may be antiquated.