what going on with apache, bind, mysql and interchange faile

[silvernetuk]

Hi,

I just got 4 email from the server to say Apache, Bind, MySQL and Interchange have failed and it will try and restart them, this time it did restart them, this has only been since a Kernel upgrade, before that the server was stable.

Where do I look to find out what going on with Apache, Bind, MySQL and Interchange failing, which logs and where are they ?

Any help would be appercaited.

Regards,
Garry

 

[steven]

Try the error_log located at /usr/local/apache/logs . Maybe you should get a server admin to help you.

-Steven

 

[silvernetuk]

Hi,

That error log is that the main error log for the whole server ? that has all the html errors in etc... ?

Ah it is, just going to download it to my desktop then look at it.
A server admin installed the kernel for me.

Regards,
Garry

 

[dgbaker]

Another to look at more logs is in /var/log

Look at the following logs;
messages
boot.log
dmesg
chkservd.log

These may give some hints.

 

[silvernetuk]

Hi,

Right I have look in

messages (/var/log)
==============
and found
Jan 20 22:49:29 silver rc.sysinit: Checking filesystems succeeded
Jan 20 22:49:30 silver rc.sysinit: Mounting local filesystems: succeeded
Jan 20 22:49:30 silver rc.sysinit: Enabling local filesystem quotas: succeeded
Jan 20 22:49:31 silver rc.sysinit: Enabling swap space: succeeded
Jan 20 22:49:33 silver init: Entering runlevel: 3
Jan 20 22:49:33 silver kudzu: Updating /etc/fstab succeeded
Jan 20 22:50:15 silver kudzu: failed
Jan 20 22:50:15 silver kudzu: Hardware configuration timed out.
Jan 20 22:50:15 silver kudzu: Run '/usr/sbin/kudzu' from the command line to re-detect.
Jan 20 22:50:15 silver sysctl: net.ipv4.ip_forward = 0
Jan 20 22:50:15 silver sysctl: net.ipv4.conf.default.rp_filter = 1
Jan 20 22:50:15 silver sysctl: kernel.sysrq = 0
Jan 20 22:50:15 silver sysctl: kernel.core_uses_pid = 1
Jan 20 22:50:15 silver network: Setting network parameters: succeeded
Jan 20 22:50:15 silver network: Bringing up loopback interface: succeeded
Jan 20 22:50:18 silver network: Bringing up interface eth0: succeeded
Jan 20 22:50:18 silver filelimits: Increasing file system limits succeeded
Jan 20 22:50:18 silver ipaliases: Bringing up eth0:1 succeeded
Jan 20 22:50:18 silver ipaliases: Routing 64.xx.xxx.88 succeeded
Jan 20 22:50:18 silver ipaliases: Bringing up eth0:2 succeeded
Jan 20 22:50:18 silver ipaliases: Routing 64.xx.xxx.89 succeeded
Jan 20 22:50:18 silver ipaliases: Bringing up eth0:3 succeeded
Jan 20 22:50:18 silver ipaliases: Routing 64.xx.xxx.54 succeeded
Jan 20 22:50:18 silver ipaliases: Bringing up eth0:4 succeeded
Jan 20 22:50:18 silver ipaliases: Routing 69.xx.xx.114 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:5 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.115 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:6 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.116 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:7 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.117 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:8 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.118 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:9 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.119 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:10 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.120 succeeded
Jan 20 22:50:19 silver ifconfig: SIOCSIFBRDADDR: Cannot assign requested address
Jan 20 22:50:19 silver ifconfig: SIOCSIFFLAGS: Cannot assign requested address
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:11 failed
Jan 20 22:50:19 silver route: SIOCADDRT: No such device
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.x.120 failed
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:12 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.113 succeeded
Jan 20 22:50:24 silver xinetd: xinetd startup succeeded
Jan 20 22:50:25 silver antirelayd: antirelayd startup succeeded
Jan 20 22:50:26 silver chkservd: chkservd startup succeeded
Jan 20 22:50:27 silver cpaneld: ==& Checking Permissions on /dev/null...Done
Jan 20 22:50:27 silver cpaneld: ==& Loading Config ....RS VALIASDIR VFILTERDIR access_log adminuser allowparkonothers allowremotedomains allowunregistereddomains basename cycle docroot dumplogs engine enginepl engineroot extracpus interchangever keeplogs loadthreshold myname port product root skipanalog skipawstats skiphorde skipneomail skipspamassassin skipwebalizer stats_log version
Jan 20 22:50:28 silver cpaneld: ==& Loading Config ....RS VALIASDIR VFILTERDIR access_log adminuser allowparkonothers allowremotedomains allowunregistereddomains basename cycle docroot dumplogs engine enginepl engineroot extracpus interchangever keeplogs loadthreshold myname port product root skipanalog skipawstats skiphorde skipneomail skipspamassassin skipwebalizer stats_log version
Jan 20 22:50:28 silver cpaneld: ==& cpaneld(logger) 5.3 Starting (1 processor system).....
Jan 20 22:50:28 silver cpaneld: ==& cpaneld 5.3 Starting ( processor system).....
Jan 20 22:50:28 silver cpanel3: cpaneld startup succeeded
Jan 20 22:50:28 silver webmaild: ==& Loading Config ....RS VALIASDIR VFILTERDIR access_log adminuser allowparkonothers allowremotedomains allowunregistereddomains basename cycle docroot dumplogs engine enginepl engineroot extracpus interchangever keeplogs loadthreshold myname port product root skipanalog skipawstats skiphorde skipneomail skipspamassassin skipwebalizer stats_log version
Jan 20 22:50:28 silver cpanel3: webmaild startup succeeded
Jan 20 22:50:31 silver cpanel3: cppop startup succeeded
Jan 20 22:50:31 silver entropychat: Listening on port 2084...
Jan 20 22:50:31 silver cpanel3: entropychat startup succeeded
Jan 20 22:50:31 silver cpanel3: startmelange startup succeeded
Jan 20 22:50:38 silver cpanel3: startinterchange startup succeeded
Jan 20 22:50:38 silver stunnel[1214]: stunnel 4.02 on i686-pc-linux-gnu PTHREAD+LIBWRAP with OpenSSL 0.9.6b [engine] 9 Jul 2001
Jan 20 22:50:38 silver stunnel[1214]: FD_SETSIZE=1024, file ulimit=1024 -& 500 clients allowed
Jan 20 22:50:38 silver cpanel3: startstunnel startup succeeded
Jan 20 22:50:39 silver whostmgrd: ==& Web Host Manager 5.3.0 Server Starting.....
Jan 20 22:50:39 silver cpanel3: whostmgrd startup succeeded
Jan 20 22:50:39 silver exim: exim startup succeeded
Jan 20 22:50:39 silver exim: antirelayd startup succeeded
Jan 20 22:50:39 silver named: named shutdown failed
Jan 20 22:50:40 silver named: named shutdown failed
Jan 20 22:50:41 silver exim: spamd startup succeeded
Jan 20 22:50:41 silver named[1333]: starting BIND 9.2.1 -u named
Jan 20 22:50:41 silver named[1333]: using 1 CPU
Jan 20 22:50:41 silver named: named startup succeeded
Jan 20 22:50:42 silver named[1337]: loading configuration from '/etc/named.conf'
Jan 20 22:50:42 silver named[1337]: no IPv6 interfaces found
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface lo, 127.0.0.1#53
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0, 69.xx.x.120#53
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0:1, 64.xx.xxx.88#53
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0:3, 64.xx.xxx.54#53
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0:4, 69.xx.xx.114#53
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0:2, 64.xx.xxx.89#53
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0:5, 69.xx.xx.115#53
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0:6, 69.xx.xx.116#53
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0:7, 69.xx.xx.117#53
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0:8, 69.xx.xx.118#53
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0:9, 69.xx.xx.119#53
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0:10, 69.xx.xx.120#53
Jan 20 22:50:42 silver crond: crond startup succeeded
Jan 20 22:50:42 silver named[1337]: listening on IPv4 interface eth0:12, 69.xx.xx.113#53
Jan 20 22:50:42 silver named[1337]: command channel listening on 127.0.0.1#953
Jan 20 22:50:42 silver anacron: anacron startup succeeded

boot.log (/var/log)
=============
and found
Jan 20 22:49:33 silver kudzu: Updating /etc/fstab succeeded
Jan 20 22:50:15 silver kudzu: failed
Jan 20 22:50:15 silver kudzu: Hardware configuration timed out.
Jan 20 22:50:15 silver kudzu: Run '/usr/sbin/kudzu' from the command line to re-detect.
Jan 20 22:50:15 silver sysctl: net.ipv4.ip_forward = 0
Jan 20 22:50:15 silver sysctl: net.ipv4.conf.default.rp_filter = 1
Jan 20 22:50:15 silver sysctl: kernel.sysrq = 0
Jan 20 22:50:15 silver sysctl: kernel.core_uses_pid = 1
Jan 20 22:50:15 silver network: Setting network parameters: succeeded
Jan 20 22:50:15 silver network: Bringing up loopback interface: succeeded
Jan 20 22:50:18 silver network: Bringing up interface eth0: succeeded
Jan 20 22:50:18 silver filelimits: Increasing file system limits succeeded
Jan 20 22:50:18 silver ipaliases: Bringing up eth0:1 succeeded
Jan 20 22:50:18 silver ipaliases: Routing 64.xx.xxx.88 succeeded
Jan 20 22:50:18 silver ipaliases: Bringing up eth0:2 succeeded
Jan 20 22:50:18 silver ipaliases: Routing 64.xx.xxx.89 succeeded
Jan 20 22:50:18 silver ipaliases: Bringing up eth0:3 succeeded
Jan 20 22:50:18 silver ipaliases: Routing 64.xx.xxx.54 succeeded
Jan 20 22:50:18 silver ipaliases: Bringing up eth0:4 succeeded
Jan 20 22:50:18 silver ipaliases: Routing 69.xx.xx.114 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:5 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.115 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:6 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.116 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:7 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.117 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:8 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.118 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:9 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.119 succeeded
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:10 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.x.120 succeeded
Jan 20 22:50:19 silver ifconfig: SIOCSIFBRDADDR: Cannot assign requested address
Jan 20 22:50:19 silver ifconfig: SIOCSIFFLAGS: Cannot assign requested address
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:11 failed
Jan 20 22:50:19 silver route: SIOCADDRT: No such device
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.x.120 failed
Jan 20 22:50:19 silver ipaliases: Bringing up eth0:12 succeeded
Jan 20 22:50:19 silver ipaliases: Routing 69.xx.xx.113 succeeded
Jan 20 22:50:24 silver xinetd: xinetd startup succeeded
Jan 20 22:50:25 silver antirelayd: antirelayd startup succeeded
Jan 20 22:50:26 silver chkservd: chkservd startup succeeded
Jan 20 22:50:27 silver cpaneld: ==& Checking Permissions on /dev/null...Done
Jan 20 22:50:27 silver cpaneld: ==& Loading Config ....RS VALIASDIR VFILTERDIR access_log adminuser allowparkonothers allowremotedomains allowunregistereddomains basename cycle docroot dumplogs engine enginepl engineroot extracpus interchangever keeplogs loadthreshold myname port product root skipanalog skipawstats skiphorde skipneomail skipspamassassin skipwebalizer stats_log version
Jan 20 22:50:28 silver cpaneld: ==& Loading Config ....RS VALIASDIR VFILTERDIR access_log adminuser allowparkonothers allowremotedomains allowunregistereddomains basename cycle docroot dumplogs engine enginepl engineroot extracpus interchangever keeplogs loadthreshold myname port product root skipanalog skipawstats skiphorde skipneomail skipspamassassin skipwebalizer stats_log version
Jan 20 22:50:28 silver cpaneld: ==& cpaneld(logger) 5.3 Starting (1 processor system).....
Jan 20 22:50:28 silver cpaneld: ==& cpaneld 5.3 Starting ( processor system).....
Jan 20 22:50:28 silver cpanel3: cpaneld startup succeeded
Jan 20 22:50:28 silver webmaild: ==& Loading Config ....RS VALIASDIR VFILTERDIR access_log adminuser allowparkonothers allowremotedomains allowunregistereddomains basename cycle docroot dumplogs engine enginepl engineroot extracpus interchangever keeplogs loadthreshold myname port product root skipanalog skipawstats skiphorde skipneomail skipspamassassin skipwebalizer stats_log version
Jan 20 22:50:28 silver cpanel3: webmaild startup succeeded
Jan 20 22:50:31 silver cpanel3: cppop startup succeeded
Jan 20 22:50:31 silver entropychat: Listening on port 2084...
Jan 20 22:50:31 silver cpanel3: entropychat startup succeeded
Jan 20 22:50:31 silver cpanel3: startmelange startup succeeded
Jan 20 22:50:38 silver cpanel3: startinterchange startup succeeded
Jan 20 22:50:38 silver cpanel3: startstunnel startup succeeded
Jan 20 22:50:39 silver whostmgrd: ==& Web Host Manager 5.3.0 Server Starting.....
Jan 20 22:50:39 silver cpanel3: whostmgrd startup succeeded
Jan 20 22:50:39 silver exim: exim startup succeeded
Jan 20 22:50:39 silver exim: antirelayd startup succeeded
Jan 20 22:50:39 silver named: named shutdown failed
Jan 20 22:50:40 silver named: named shutdown failed
Jan 20 22:50:41 silver exim: spamd startup succeeded
Jan 20 22:50:41 silver named: named startup succeeded
Jan 20 22:50:42 silver crond: crond startup succeeded
Jan 20 22:50:42 silver anacron: anacron startup succeeded
Jan 20 22:50:42 silver bandmin: Starting bandmin:
Jan 20 22:50:43 silver bandmin: iptables v1.2.5: Couldn't load target `acctboth':/lib/iptables/libipt_acctboth.so: cannot open shared object file: No such file or directory
Jan 20 22:50:43 silver bandmin:
Jan 20 22:50:43 silver bandmin: Try `iptables -h' or 'iptables --help' for more information.
Jan 20 22:50:43 silver bandmin: iptables v1.2.5: Couldn't load target `acctboth':/lib/iptables/libipt_acctboth.so: cannot open shared object file: No such file or directory
Jan 20 22:50:43 silver bandmin:
Jan 20 22:50:43 silver bandmin: Try `iptables -h' or 'iptables --help' for more information.
Jan 20 22:50:44 silver bandmin:
Jan 20 22:50:44 silver rc: Starting bandmin: succeeded
Jan 20 22:50:44 silver portsentry: Starting portsentry -tcp: succeeded
Jan 20 22:50:44 silver portsentry: Starting portsentry -udp: succeeded
Jan 20 23:06:34 silver filelimits: Increasing file system limits succeeded

dmesg (/var/log)
============
Everything look ok in here

chkservd.log (/var/log)
================
and found
[Mon Jan 20 22:44:46 2003] Service check ....apache [+]...bind [+]...exim [+]...ftpd [+]...imap [+]...interchange [+]...mysql [+]...syslogd [+]...webmail [+]...Done
Loading services .....interchange....apache....exim....webmail....mysql....ftpd....syslogd....imap....bind..
[Mon Jan 20 22:50:26 2003] Service check ....apache [-Restarting apache....
system: /scripts/checkmaxclients
system: killall -TERM httpd
system: sleep 2
system: killall -9 httpd
system: /etc/rc.d/init.d/httpd stop
system: /etc/rc.d/init.d/httpd startssl
system: /usr/local/apache/bin/apachectl startssl
system: /etc/rc.d/init.d/httpd start
system: /usr/local/apache/bin/apachectl start
]...bind [-Restarting bind....
system: /scripts/restartsrv bind
]...exim [+]...ftpd [+]...imap [+]...interchange [-Restarting interchange....
system: /usr/local/cpanel/bin/startinterchange
]...mysql [-Restarting mysql....
system: /scripts/restartsrv mysql
]...syslogd [+]...webmail [+]...Done
[Mon Jan 20 23:01:38 2003] Service check ....apache [+]...bind [+]...exim [+]...ftpd [+]...imap [+]...interchange [+]...mysql [+]...syslogd [+]...webmail [+]...Done

error_log (/usr/local/apache/logs)
========================
and found
[Mon Jan 20 22:50:38 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:38 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:38 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:38 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:38 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:38 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:38 2003] [warn] pid file /usr/local/apache/logs/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Mon Jan 20 22:50:39 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:39 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:39 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:39 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:39 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:39 2003] [warn] NameVirtualHost 69.10.8.120:80 has no VirtualHosts
[Mon Jan 20 22:50:39 2003] [crit] (98)Address already in use: make_sock: could not bind to port 443
[Mon Jan 20 22:50:39 2003] [notice] Apache/1.3.27 (Unix) mod_log_bytes/0.3 mod_bwlimited/1.0 PHP/4.2.3 FrontPage/5.0.2.2510 mod_ssl/2.8.11 OpenSSL/0.9.6b configured -- resuming normal operations
[Mon Jan 20 22:50:39 2003] [notice] suEXEC mechanism enabled (wrapper: /usr/local/apache/bin/suexec)
[Mon Jan 20 22:50:39 2003] [notice] Accept mutex: sysvsem (Default: sysvsem)

I can see stuff failed above which was when Bind and Apache, MySQL and interchange crashed out

But can anyone see why or what made it crash out ?
Any help very much appercaited

Regards,
Garry

 

[silvernetuk]

Hi,

Can anyone tell me what I should be looking for ?

Regards,
Garry

 

[dgbaker]

Do you have Bastille Firewall on your machine?

We have found that if Bastille is having issues with iptables or ipchains, it can cause some flaky things, like services restarting (trying to) every few minutes, and site response is down to a crawl.

 

[silvernetuk]

Hi,

No I don't have Bastille Firewall on my machine.

Regards,
Garry

 

[silvernetuk]

Hi,

It look like the server was re-booted.

Regars,
Garry

 

[annualhost]

Garry,

I have been having the same problem on my box as well. I am however running Bastille. But it was running fine until about two weeks ago.

I recognize your name from the @webhost forums... I have my box with them as well. That makes me even more curious.

At first I thought my box was hacked but after further investigation I can't seem to find any evidence of that.

Any ideas would be helpful...

Thanks,

Nathan

 

[dgbaker]

Any recent kernel updates? That's what broke Bastille on one of our servers, and caused a lot of issue when it tried to start up.

 

[annualhost]

Garry,

I have been having the same problem on my box as well. I am however running Bastille. But it was running fine until about two weeks ago.

I recognize your name from the @webhost forums... I have my box with them as well. That makes me even more curious.

At first I thought my box was hacked but after further investigation I can't seem to find any evidence of that.

Any ideas would be helpful...

Thanks,

Nathan

 

[silvernetuk]

Hi,

I think your problem with Bastille is because we had a Kernel upgrade or I know I did and a lot of other people did and as David pointed out, that what broke Bastille on one of his servers. So maybe that the same problem for you.

I also know the servers was rebooted that night I had those emails.

Regards,
Garry

 

[dgbaker]

Garry;

After your server rebooted, are things staying up? If not and you want, PM me and maybe we can help you by taking a look at your server. See our consulting site http://www.dgbaker.com

 

[silvernetuk]

Hi,

Yes everything is back up and back to normal

Regards,
Garry