cPanel services failing: Segfaults

[paulkoan]

Hello - my cpanel services are failing due to segfaults. This started happening at midnight UTC, which is when cpanel update runs.

The dmesg and messages just indicate a segfault, but with no additional info. Can anyone offer any diagnoses tips?

=====================================================
[root@cpanel ~]# sudo /etc/init.d/cpanel restart                                                 
Stopping tailwatchd:                                                                             
!!                                                                                               
ATTENTION ATTENTION ATTENTION ATTENTION                                                          

Cpanel::TailWatch::Eximstats appears to have unprocessed SQL in /var/cpanel/sql/eximstats.sql.

When mysql is unable to execute a query they are logged for processing later.

Eventually these SQL files may be handled automatically and this message will not appear.

In the meantime you can execute the queries as root with something like this:

  mv /var/cpanel/sql/eximstats.sql /var/cpanel/sql/eximstats.sql.tmp_working_copy
  /usr/local/cpanel/libexec/tailwatchd --restart                                
  mysql eximstats < /var/cpanel/sql/eximstats.sql.tmp_working_copy              

Once you are sure all is well you can remove /var/cpanel/sql/eximstats.sql.tmp_working_copy


ATTENTION ATTENTION ATTENTION ATTENTION
!!                                    

2009-04-01 03:13:32 UTC [main] Current process '4930' stopped
                                                           [  OK  ]
Stopping cPanel services:  /etc/init.d/functions: line 489: 14005 Segmentation fault      (core dumped) "$@"
                                                           [FAILED]                                        
Stopping cPanel dav services:  /etc/init.d/functions: line 489: 14006 Segmentation fault      (core dumped) "$@"
                                                           [FAILED]                                            
Stopping cPanel brute force detector services:             [  OK  ]                                            
Stopping pop3 services:  Waiting for cppop to shutdown ... ...Done                                             
Waiting for cppop-ssl to shutdown ... ...Done                                                                  
                                                           [  OK  ]                                            
Stopping cPanel log services:                              [  OK  ]                                            
Stopping cPanel Chat services:                             [  OK  ]                                            
Stopping Melange Chat services:                            [FAILED]                                            
Stopping InterChange services:                             [  OK  ]                                            
Stopping cPanel ssl services:                              [  OK  ]                                            
Stopping mailman services:  /etc/init.d/functions: line 489: 14043 Segmentation fault      (core dumped) "$@"  
                                                           [FAILED]                                            
Starting cPanel services:                                  [  OK  ]                                            
Starting cPanel brute force detector services:             [  OK  ]                                            
Starting cPanel dav services: /bin/bash: line 1: 14065 Segmentation fault      /usr/local/cpanel/etc/init/startcpdavd
                                                           [FAILED]                                                 
Starting cPanel Log services: /bin/bash: line 1: 14070 Segmentation fault      /usr/local/cpanel/cpanellogd         
                                                           [FAILED]
Starting pop3 services:                                    [  OK  ]
Starting cPanel Chat services: Listening on port 2084...
                                                           [  OK  ]
Starting Melange Chat services:
Starting InterChange services:                             [  OK  ]
Starting cPanel ssl services: /bin/bash: line 1: 14095 Segmentation fault      /usr/local/cpanel/startstunnel
                                                           [FAILED]
Starting mailman services:  Starting Mailman's master qrunner.
                                                           [  OK  ]
Starting tailwatchd:
!!
ATTENTION ATTENTION ATTENTION ATTENTION

Cpanel::TailWatch::Eximstats appears to have unprocessed SQL in /var/cpanel/sql/eximstats.sql.

When mysql is unable to execute a query they are logged for processing later.

Eventually these SQL files may be handled automatically and this message will not appear.

In the meantime you can execute the queries as root with something like this:

  mv /var/cpanel/sql/eximstats.sql /var/cpanel/sql/eximstats.sql.tmp_working_copy
  /usr/local/cpanel/libexec/tailwatchd --restart
  mysql eximstats < /var/cpanel/sql/eximstats.sql.tmp_working_copy

Once you are sure all is well you can remove /var/cpanel/sql/eximstats.sql.tmp_working_copy


ATTENTION ATTENTION ATTENTION ATTENTION
!!

[Wed Apr  1 03:13:51 2009] Starting /usr/local/cpanel/libexec/tailwatchd daemon
Log is at /usr/local/cpanel/logs/tailwatchd_log

=====================================================

[paulkoan]

After a upcp --force, I cleared most of the segfaults.

However I am still getting segfaults from dnsqueue, from the cronjob:

/bin/sh: line 1:  8621 Segmentation fault      /usr/local/cpanel/whostmgr/bin/dnsqueue > /dev/null 2>&1

Any ideas?

[myce]

I am having the same or similar problems. First, I tried to restart cPanel, and it went into an auto-reinstall, generating 21,784 of recovery before my ssh session lost the session (don't know why; it didn't disconnect; but it stopped while processing rrdtool-1.3.6). So, I restarted again and got this :

root@server [/]# /etc/init.d/cpanel restart
Stopping tailwatchd:  2009-04-02 08:13:45 UTC [main] Current process '443' stopped
                                                           [  OK  ]
Stopping cPanel services:  Waiting for cpsrvd to shutdown ... ...Done
Waiting for cpsrvd-ssl to shutdown ... ....Done
                                                           [  OK  ]
Stopping cPanel dav services:  Can't locate XML/LibXML.pm in @INC (@INC contains: /usr/local/cpanel/Cpanel/CPAN/overload/__File /usr/local/cpanel /usr/lib/perl5/site_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.7/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.6/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.7/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.6/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.5/i386-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib/perl5/5.8.8/i386-linux-thread-multi /usr/lib/perl5/5.8.8 .) at /usr/lib/perl5/site_perl/5.8.8/Net/DAV/Server.pm line 14.
BEGIN failed--compilation aborted at /usr/lib/perl5/site_perl/5.8.8/Net/DAV/Server.pm line 14.
Compilation failed in require at /usr/local/cpanel/Cpanel/NetDAVServer.pm line 18.
Compilation failed in require at /usr/local/cpanel/libexec/cpdavd line 14.
BEGIN failed--compilation aborted at /usr/local/cpanel/libexec/cpdavd line 14.
Waiting for cpdavd to shutdown ... ...Done
Waiting for cpdavd-ssl to shutdown ... ...Done
                                                           [  OK  ]
Stopping cPanel brute force detector services:  Waiting for cphulkd.pl to shutdown ... ...Done
Waiting for cPhulkd to shutdown ... ...Done
Waiting for cphulkd to shutdown ... ...Done
                                                           [  OK  ]
Stopping pop3 services:  Waiting for cppop to shutdown ... ...Done
Waiting for cppop-ssl to shutdown ... ...Done
                                                           [  OK  ]
Stopping cPanel log services:                              [  OK  ]
Stopping cPanel Chat services:                             [  OK  ]
Stopping Melange Chat services:                            [FAILED]
Stopping InterChange services:                             [FAILED]
Stopping cPanel ssl services:                              [  OK  ]
Stopping mailman services:  Shutting down Mailman's master qrunner
mailmanctl: no process killed
                                                           [FAILED]
Starting cPanel services:                                  [  OK  ]
Starting cPanel brute force detector services:             [  OK  ]
Starting cPanel dav services:                              [FAILED]
Starting cPanel Log services: ==> cPanel Log Daemon version 24.0
                                                           [  OK  ]
Starting pop3 services: Waiting for cppop to shutdown ... ...Done
Waiting for cppop-ssl to shutdown ... ...Done
                                                           [  OK  ]
Starting cPanel Chat services: Listening on port 2084...
                                                           [  OK  ]
Starting Melange Chat services:                            [  OK  ]
Starting cPanel ssl services: Melange Chat Server (Version 1.10), Apr-25-1999
(C) 1998,1999 by Christian Walter, All rights reserved
http://melange.terminal.at       Email: chris@terminal.at

Unable to open logfile /var/log/melange.log - Logging disabled.
--> Administrator Features: Disabled
--> System Messages: Off
--> User Channels: Enabled
--> Anonymous Channel: Enabled
--> Only unique nicknames allowed: No
--> Guest login allowed: Yes
--> Users are allowed to change their nicknames: Yes
--> Default Security Policy: ALLOW ALL HOSTS
--> List of hosts NOT allowed to connect:
--> Default BannTime Setting [sec]: 3600
--> Max Idle Time [sec]: 0
--> LogLevel: 3
--> ServerPort: 6666
--> Message Logfile: disabled
--> Logging Activated: 0
melange server bind failure 98: Address already in use
Using Native SSL support (stunnel not needed)
                                                           [  OK  ]
Starting mailman services:  Starting Mailman's master qrunner.
                                                           [  OK  ]
Starting tailwatchd: [Thu Apr  2 04:14:06 2009] Starting /usr/local/cpanel/libexec/tailwatchd daemon
Log is at /usr/local/cpanel/logs/tailwatchd_log
                                                           [  OK  ]
root@server [/]#
root@server [/]#

I tried to do another forced upcp, but if failed with:

Internal Server Error
      Premature end of script headers: /usr/local/cpanel/whostmgr/docroot/cgi/live_upcp.cgi: Please check   /  usr /  local /  cpanel /  logs /  error_log  for the exact error.
      
________________________________________
cpsrvd/11.24 Server at server.alcme.net
________________________________________

There is nothing in /usr/local/cpanel/logs/error_log except:

2009-04-02 08:14:06 info [tailwatchd] BSD::Resource not available. Resource Limits not adjusted

I am getting "Premature end of script headers" on almost everything I try to run. I am running "Current":

cPanel 11.24.4-C34960 - WHM 11.24.2 - X 3.9
CENTOS 5.3 i686 standard on server

Any advice for me too?

[myce]

Well, I finally got mine back up by killing a "rogue" upcp process, installing Perl 5.8.8, and then running /scripts/upcp --force. Perl had downgraded during the cPanel update last night (I saw that in the logs), and getting Perl fixed got rid of the "Premature end of script headers" so that upcp would run.

1) Kill rogue upcp process
2) /scripts/fixeverything
3) wget http://layer1.cpanel.net/perl588installer.tar.gz
4) tar xfvz perl588installer.tar.gz
5) cd perl588installer
6) ./install
7) /usr/local/cpanel/bin/checkperlmodules
8) /scripts/upcp --force
9) /etc/init.d/cpanel restart

Now, everything appears to be working! Hope it works for you too! Good luck!

[crazyaboutlinux]

don't forget to restart cpanel after upgrade cPanel

service cpanel restart

Nilesh

[myce]

Good point! I added a restart!

[Silent Ninja]

I don't know if this is your case, but on Argentina, all the Fibertel dns are being blocked by cPanel's fault.

You have to use international dns for all cpanel and scripts to work.
This is my new resolv.conf (it works now because of this change)

nameserver 4.2.2.4
nameserver 4.2.2.2
nameserver 4.2.2.3
nameserver 4.2.2.1

This DNS's are from Verizon, but you can also use OpenDNS which works perfectly too.

[chandro]

i think all is fixed, i had that problems about services failing, but i modify the quotas of the VPS and i dont see any error now. lets wait....

[egillette]

Yeah, I had the same problem with a client's machine.

Just in case anyone is interested in using the OpenDNS servers, here are the IP's for you resolv.conf:

208.67.220.220
208.67.222.222
208.67.220.222

Hope this helps. . .

[InterServed]

Very old topic and I'm unsure if the issue I've just discovered is perfectly related to this topic but I'll post it in here < asking a mod to move it on a separate topic if it's the case >.

Apr 11 00:16:49 nlsrv1 kernel: cpanellogd[6832]: segfault at 00002ba87b188340 rip 00002ba87b188340 rsp 00007fffa1f1c728 error 15
Apr 11 00:25:09 nlsrv1 kernel: cpanellogd[12646]: segfault at 00002ba87b188340 rip 00002ba87b188340 rsp 00007fffa1f1c728 error 15
Apr 11 00:33:29 nlsrv1 kernel: cpanellogd[17824]: segfault at 00002ba87b188340 rip 00002ba87b188340 rsp 00007fffa1f1c728 error 15

I started to notice this problems since updated to centos 5.6

[todobravo]

I have the same problem on my server since the updgrade of Centos 5.6
Apr 14 09:18:51 ns355888 kernel: cpanellogd[22395]: segfault at 42484280 ip 42484280 sp bf829668 error 15 in ld-2.5.so (deleted)[42484000+1000]

Did you find a solution ?

[todobravo]

Here the solution with the help of Cpanel Support :
1) cd /root
2) wget http://layer1.cpanel.net/perl588installer.tar.gz
3) tar xfvz perl588installer.tar.gz
4) cd perl588installer
5) ./install
6) /scripts/checkperlmodules --full --force
7) /scripts/upcp --force
8) mv /usr/local/cpanel/3rdparty/bin/webalizer /usr/local/cpanel/3rdparty/webalizer.bak
9) mv /usr/local/cpanel/3rdparty/bin/english /usr/local/cpanel/3rdparty/bin/english.bak
10) /usr/local/cpanel/bin/buildwebalizer english
11) /etc/init.d/cpanel restart

[egillette]

Nice one. . .that's exactly what I did, but it seems that you posted the solution before I came back to post it.

Regardless, at least if someone else runs into the same snag your post will help them to fix it, which is the whole point of this thread. =0)