Problems since last night's update?

dspillett

Active Member
Oct 2, 2005
26
0
151
My cPanel setup upgraded itself at approx 0500GTM (it is currently 1025GMT) and since then I'm seeing the following problems:
  • The load reading has been constantly higer than usual, even though the machine is practically idle most of the time. Running top shows no process using anything more than a fraction of CPU time, in eith busy or io-wait states. /proc/io_status confirms that there is little IO going on and the load readings of the host machine confirm that there is little CPU being used. I'd notmall expect to see load readings between 0.2 and 0.5 on this machine except the occasional burst of activity, and it is currently constantly above 3.5 despite apparently no tasks doing anything.
  • Some services keep failing, or at least the monitor things they do and keeps restarting them. This is usually exim and cpservd. Exim does seem to be running OK, as I'm getting the "service automatically restarted" messages to an external account.
  • WHM keeps stopping responding to HTTPS requests on port 2087, which lends some credence to the messages that cpservd is stopping, but it doesn't stop responding to HTTP requests on port 2086.
  • When poking arouns WHM over HTTP://<addr>:2086, the I keep seemingly getting logged out (so the browsers HTTP-Auth dialogue appears). Once I've re-authenticated I continue where I left off, but twice this has happened so many times in a short time that the brute-force detector blocks me for a few minutes.
This instance of cPanel/RELEASE in running in a UserModeLinux based VM on CentOS 4.5.


Is anyone else having similar trouble, or is it just me?

Is there anywhere else I can check for what might be causing load to show higher values than it should when the machine it idle?

My first worry was that the VM had been hacked, but as the problems seem to have started directly after the update (that is when I started getting the "service failed, restarting" emails I'm not currently thinking that this is the case. I don't think the unexpected load reading is due it UML as my UML kernel has not changed recently and has been running fine for months.

I've tried forcing a complete update of cPanel (in case something odd happened in the overnight update that left a problem a forced update would clear), making sure that the underlying CentOS is uptodate, and rebooting - all several times.
 

ahostli

Active Member
May 28, 2006
40
0
156
No (maybe because Im using STABLE release), but after yerterdays updates im getting this error when trying to use phpMyadmin:
/usr/local/cpanel/3rdparty/bin/php: /usr/lib/mysql/libmysqlclient.so.14: no version information available (required by /usr/local/cpanel/3rdparty/bin/php) /usr/local/cpanel/3rdparty/bin/php: relocation error: /usr/local/cpanel/3rdparty/bin/php: symbol __udivdi3, version libmysqlclient_14 not defined in file libmysqlclient.so.14 with link time reference
 

dspillett

Active Member
Oct 2, 2005
26
0
151
No (maybe because Im using STABLE release)
If I can't sort out the current issues quickly any other way, is downgrading from RELEASE to STABLE considered a safe operation? (though is you are having mysql problems on STABLE, maybe I'd just be swapping one problem for another...)
 

nyjimbo

Well-Known Member
Jan 25, 2003
1,135
1
168
New York
You might have something going on unrelated to the update/upgrade.

Do a bunch of things like look for weird perl tasks:

ps ax | grep "perl"

Look for lots of ftp login attempts:

pure-ftpwho

look for excessive pop logins:

ps ax| grep "pop"

How many tasks are running in top, even if they show very little cpu usage each?

Do a "netstat -m" to see if you are getting slammed on your network with anything that might not show up on top or other checks. If your mbuf usage is very high then some kind of attack could be happening.

What does WHM show when you look for "Show mysql processes" and "Apache Status", anything weird going on.

Did you try restarting some of the monsters like apache, exim, mysql, etc.. sometimes something might be stuck in a loop.

Check "top" to see if you are running out of ram and doing alot of swapping.

Do a "exiwhat | sort +6| more" to do a quick check to see if you are getting hammered by these new zombie spammers

there is probably alot more you can check but some of these might give you a clue if anything weird is running that might be causing this load issue.
 

dspillett

Active Member
Oct 2, 2005
26
0
151
You might have something going on unrelated to the update/upgrade.
Though the the fact the problems started immediately after the update, or so it would seem, is bit of a coincidence.

Do a bunch of things like <snip>
None of those, or my earlier scans for ssh/ftp/exim/apache logs showed anything out of the ordinary.

Do a "netstat -m" to see if you are getting slammed on your network with anything that might not show up on top or other checks. If your mbuf usage is very high then some kind of attack could be happening.
-m does not seem to be an option on netstat. I'm fairly sure that it isn't network activity, as I'm not seeing lots of activity going through the host machine.

Did you try restarting some of the monsters like apache, exim, mysql, etc.. sometimes something might be stuck in a loop.
None of those processes seem to be churning at all, and I've tried completely rebooting the VM as well as restarting the individual services.

Check "top" to see if you are running out of ram and doing alot of swapping.
73 tasks, which I don't think is abnormal, none of which are showing much activity in top, and the machine isn't yet touching swap since the last reboot a few hours ago.

Starting artificial tasks a short time apart and checking the PIDs allocated shows that processes don't seem to be being created and destroyed faster than top will see them.[/QUOTE]

Whatever is happening is affecting secure variants of the services more than the insecure ones, as getting into http:2082 is far more reliable than getting into https:2083 (it works almost all the time rather than almost, but not quite, never). The same goes for the WHM and Webmail ports. On the http ports you do regularly need to re-authenticate, but I'm thinking that this is due to the precess monitor seeing the https version not responding and resetting the whole lot.

Luckyly, Apache and the IMAP/POP services seem to be functioning OK, so people aren't losing mail and page views.
 
Last edited:

nyjimbo

Well-Known Member
Jan 25, 2003
1,135
1
168
New York
-m does not seem to be an option on netstat. I'm fairly sure that it isn't network activity, as I'm not seeing lots of activity going through the host machine.
Ouch, I guess thats a freebsd switch. I hate it when these O/S's have different syntax.
 

dspillett

Active Member
Oct 2, 2005
26
0
151
Ouch, I guess thats a freebsd switch. I hate it when these O/S's have different syntax.
Aye. BSD and the tool variant usually found on Linux systems don't differ much, though sometimes they do in unexpected places.

Could you send me the text describing the option from your system's man page? There is likely another tool in the standard set that does the same job under Linux (or even just a /proc entry that carries the appropriate data).
 

nyjimbo

Well-Known Member
Jan 25, 2003
1,135
1
168
New York
Could you send me the text describing the option from your system's man page? ..
NAME
netstat -- show network status

DESCRIPTION
The netstat command symbolically displays the contents of various net-
work-related data structures. There are a number of output formats,
depending on the options for the information presented.

netstat [-AaLnSW] [-f protocol_family | -p protocol] [-M core]
[-N system]
Display a list of active sockets (protocol control blocks) for
each network protocol, for a particular protocol_family, or for a
single protocol. If -A is also present, show the address of a
protocol control block (PCB) associated with a socket; used for
debugging. If -a is also present, show the state of all sockets;
normally sockets used by server processes are not shown. If -L
is also present, show the size of the various listen queues. The
first count shows the number of unaccepted connections, the sec-
ond count shows the amount of unaccepted incomplete connections,
and the third count is the maximum number of queued connections.
If -S is also present, show network addresses as numbers (as with
-n) but show ports symbolically.

netstat -i | -I interface [-abdhntW] [-f address_family] [-M core]
[-N system]
Show the state of all network interfaces or a single interface
which have been auto-configured (interfaces statically configured
into a system, but not located at boot time are not shown). An
asterisk (``*'') after an interface name indicates that the
interface is ``down''. If -a is also present, multicast
addresses currently in use are shown for each Ethernet interface
and for each IP interface address. Multicast addresses are shown
on separate lines following the interface address with which they
are associated. If -b is also present, show the number of bytes
in and out. If -d is also present, show the number of dropped
packets. If -h is also present, print all counters in human
readable form. If -t is also present, show the contents of
watchdog timers. If -W is also present, print interface names
using a wider field size.

netstat -w wait [-I interface] [-d] [-M core] [-N system]
At intervals of wait seconds, display the information regarding
packet traffic on all configured network interfaces or a single
interface. If -d is also present, show the number of dropped
packets.

netstat -s [-s] [-z] [-f protocol_family | -p protocol] [-M core]
[-N system]
Display system-wide statistics for each network protocol, for a
particular protocol_family, or for a single protocol. If -s is
repeated, counters with a value of zero are suppressed. If -z is
also present, reset statistic counters after displaying them.

netstat -i | -I interface -s [-f protocol_family | -p protocol] [-M core]
[-N system]
Display per-interface statistics for each network protocol, for a
particular protocol_family, or for a single protocol.

netstat -m [-M core] [-N system]
Show statistics recorded by the memory management routines
(mbuf(9)). The network manages a private pool of memory buffers.

netstat -r [-AanW] [-f address_family] [-M core] [-N system]
Display the contents of all routing tables, or a routing table
for a particular address_family. If -A is also present, show the
contents of the internal Patricia tree structures; used for
debugging. If -a is also present, show protocol-cloned routes
(routes generated by an RTF_PRCLONING parent route); normally
these routes are not shown. When -W is also present, show the
path MTU for each route, and print interface names with a wider
field size.

netstat -rs [-s] [-M core] [-N system]
Display routing statistics. If -s is repeated, counters with a
value of zero are suppressed.

netstat -g [-W] [-f address_family] [-M core] [-N system]
Show information related to multicast (group address) routing.
By default, show the IP Multicast virtual-interface and routing
tables, and multicast group memberships.

netstat -gs [-s] [-f address_family] [-M core] [-N system]
Show multicast routing statistics. If -s is repeated, counters
with a value of zero are suppressed.

Some options have the general meaning:

-f address_family, -p protocol
Limit display to those records of the specified address_family or a
single protocol. The following address families and protocols are
recognized:

Family Protocols
inet (AF_INET) bdg, divert, icmp, igmp, ip, ipsec,
pim, tcp, udp
inet6 (AF_INET6) bdg, icmp6, ip6, ipsec6, rip6, tcp, udp
pfkey (PF_KEY) pfkey
atalk (AF_APPLETALK) ddp
netgraph, ng (AF_NETGRAPH) ctrl, data
ipx (AF_IPX) ipx, spx
unix (AF_UNIX)
link (AF_LINK)

-M Extract values associated with the name list from the specified
core instead of the default /dev/kmem.

-N Extract the name list from the specified system instead of the
default, which is the kernel image the system has booted from.

-n Show network addresses and ports as numbers. Normally netstat
attempts to resolve addresses and ports, and display them symboli-
cally.

-W In certain displays, avoid truncating addresses even if this causes
some fields to overflow.

The default display, for active sockets, shows the local and remote
addresses, send and receive queue sizes (in bytes), protocol, and the
internal state of the protocol. Address formats are of the form
``host.port'' or ``network.port'' if a socket's address specifies a net-
work but no specific host address. When known, the host and network
addresses are displayed symbolically according to the databases hosts(5)
and networks(5), respectively. If a symbolic name for an address is
unknown, or if the -n option is specified, the address is printed numeri-
cally, according to the address family.

The interface display provides a table of cumulative statistics regarding
packets transferred, errors, and collisions. The network addresses of
the interface and the maximum transmission unit (``mtu'') are also dis-
played.

The routing table display indicates the available routes and their sta-
tus. Each route consists of a destination host or network, and a gateway
to use in forwarding packets. The flags field shows a collection of
information about the route stored as binary choices. The individual
flags are discussed in more detail in the route(8) and route(4) manual
pages. The mapping between letters and flags is:

1 RTF_PROTO1 Protocol specific routing flag #1
2 RTF_PROTO2 Protocol specific routing flag #2
3 RTF_PROTO3 Protocol specific routing flag #3
B RTF_BLACKHOLE Just discard pkts (during updates)
b RTF_BROADCAST The route represents a broadcast address
C RTF_CLONING Generate new routes on use
c RTF_PRCLONING Protocol-specified generate new routes on use
D RTF_DYNAMIC Created dynamically (by redirect)
G RTF_GATEWAY Destination requires forwarding by intermediary
H RTF_HOST Host entry (net otherwise)
L RTF_LLINFO Valid protocol to link address translation
M RTF_MODIFIED Modified dynamically (by redirect)
R RTF_REJECT Host or net unreachable
S RTF_STATIC Manually added
U RTF_UP Route usable
W RTF_WASCLONED Route was generated as a result of cloning
X RTF_XRESOLVE External daemon translates proto to link address

Direct routes are created for each interface attached to the local host;
the gateway field for such entries shows the address of the outgoing
interface. The refcnt field gives the current number of active uses of
the route. Connection oriented protocols normally hold on to a single
route for the duration of a connection while connectionless protocols
obtain a route while sending to the same destination. The use field pro-
vides a count of the number of packets sent using that route. The inter-
face entry indicates the network interface utilized for the route.

When netstat is invoked with the -w option and a wait interval argument,
it displays a running count of statistics related to network interfaces.
An obsolescent version of this option used a numeric parameter with no
option, and is currently supported for backward compatibility. By
default, this display summarizes information for all interfaces. Infor-
mation for a specific interface may be displayed with the -I option.

The bpf(4) flags displayed when netstat is invoked with the -B option
represents the underlying parameters of the bpf peer. Each flag is repre-
sented as a single lower case letter. The mapping between the letters
and flags in order of appearance are:

p Set if listening promiscuously
i BIOCIMMEDIATE has been set on the device
f BIOCGHDRCMPLT status: source link addresses are being filled auto-
matically
s BIOCGSEESENT status: see packets originating locally and remotely on
the interface.
a Packet reception generates a signal
l BIOCLOCK status: descriptor has been locked

For more information about these flags, please refer to bpf(4).
 

nat

Well-Known Member
Jan 16, 2003
209
0
166
No (maybe because Im using STABLE release), but after yerterdays updates im getting this error when trying to use phpMyadmin:
/usr/local/cpanel/3rdparty/bin/php: /usr/lib/mysql/libmysqlclient.so.14: no version information available (required by /usr/local/cpanel/3rdparty/bin/php) /usr/local/cpanel/3rdparty/bin/php: relocation error: /usr/local/cpanel/3rdparty/bin/php: symbol __udivdi3, version libmysqlclient_14 not defined in file libmysqlclient.so.14 with link time reference
Same error for me since yesterday. Anyone find a solution to this?
 

dashi

Member
Aug 9, 2007
8
0
51
Hi,

after tomorrow update php session stop work
also some time apache cant execute php files allow only save.
please ASAP
 

cPDan

cPanel Staff
Staff member
Mar 9, 2004
724
15
243
Hi,

after tomorrow update php session stop work
also some time apache cant execute php files allow only save.
please ASAP
Without a ticket I can only guess, but try:

.php is configured
Zend is resinstaleld or removed from php.ini
 

nat

Well-Known Member
Jan 16, 2003
209
0
166
/scripts/upcp --force on stable did not fix it. Upgrading to Release from Stable fixed it for me. Now 3rd party php scripts like fantastico are not obeying /usr/local/cpanel/3rdparty/lib/php.ini, they use /usr/local/lib/php.ini instead.
 

cix

Well-Known Member
Nov 6, 2003
74
0
156
My fantastico is not working:
No input file specified. We use RELEASE version of cpanel.

Could be the same related problem?
 

nat

Well-Known Member
Jan 16, 2003
209
0
166
My fantastico is not working:
No input file specified. We use RELEASE version of cpanel.
Could be the same related problem?
dir -ald /tmp

If it is not drwxrwxrwxt, change it back to drwxrwxrwxt

chmod 1777 /tmp

Also check to insure /tmp is not full.

df -h



My problem is not a fantastico problem as it happens with all 3rd party scripts.