Weird cPanel Issue / client sites going offline

[skysel]

hi,

last few days I'm experiencing a problem with my cpanel server.

All of client websites stop responding for a few minutes, then they come back online. In that time, cpanel web interface is working.

If I restart apache manually in that time, the sites start working again. I've checked apache error log, but was unable to find any errors that may be causing this issue.

Any advice? I'm really puzzled, don't know what else to check for.

[PHP Warner]

Have you tried to build apache again?

[skysel]

I have not. Will do and report how it goes.

[skysel]

Have you tried to build apache again?

I have rebuilded apache, however the problem persists. I'm using Munin Service Monitor and during these "outages", I've noticed spikes in following areas:

- netstat - 300+ connections; average is around 50-80
- number of processes - 400+; average is around 250
- cpu usage
- commited memory usage

I'm guessing there has to be some kind of malicious software running on the system, however I've done a complete system sweep of malicious sites and fixed them all - i think. Also, I have CSF firewall installed which prevents the use of unauthorised ports.

Any other suggestions?

System info:
CPU: Quad-Core Intel(R) Xeon(R) CPU E5504 @ 2.00GHz
Memory: 12287856k/13631488k available (2575k kernel code, 285772k reserved, 1304k data, 212k init)
Kernel: 2.6.18-194.26.1.el5
Cpanel build: 11.28.64
Apache: Server Version: Apache/2.0.63 (Unix) mod_ssl/2.0.63 OpenSSL/0.9.8e-fips-rhel5 DAV/2 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635 PHP/5.2.16 mod_perl/2.0.4 Perl/v5.8.8
MySQL: Currently installed: 5.0.91
System: CentOS 5.5 x64 standard

[LinuxTechie]

Hello,

As the connections and processes are high it will be really good to do a security audit. You can install mod security, do a clamscan/maldetect scanning as a security measure. Also make sure your kernel is non vulnerable!

[cPanelTristan]

hi,

last few days I'm experiencing a problem with my cpanel server.

All of client websites stop responding for a few minutes, then they come back online. In that time, cpanel web interface is working.

If I restart apache manually in that time, the sites start working again. I've checked apache error log, but was unable to find any errors that may be causing this issue.

Any advice? I'm really puzzled, don't know what else to check for.

I'm quoting this message as it already tells me the issue isn't likely a malicious process running on the machine. You have netstat showing high connections and the issue is Apache, so you are either hitting MaxClients (so your Apache cannot serve any more connections for sites), or you are getting hit by a denial of service (DoS) attack. This is only Apache getting hammered by connections, so it isn't any other service being impacted (your cPanel is fine for example, you can log into SSH for example).

I have a post that I did recently on how to handle both of the above type of situations (MaxClients limit or DoS attack):

how to restart apache trough cpanel command line?

Thanks.

[skysel]

I'm quoting this message as it already tells me the issue isn't likely a malicious process running on the machine. You have netstat showing high connections and the issue is Apache, so you are either hitting MaxClients (so your Apache cannot serve any more connections for sites), or you are getting hit by a denial of service (DoS) attack. This is only Apache getting hammered by connections, so it isn't any other service being impacted (your cPanel is fine for example, you can log into SSH for example).

I have a post that I did recently on how to handle both of the above type of situations (MaxClients limit or DoS attack):

how to restart apache trough cpanel command line?

Thanks.

Thank you, cPanelTristan. I've made the modifications on the server, raised the MaxClients limit (there has been 1 occurence in error log that apache hit max) and also applied the iptables rule. Now I'll have to wait few days to see if it's going any better.

Meanwhile, for a better presentation, below are relevant graphs to my problems if you think you can suggest anything else:

Apache:



CPU:


INODE:


NETSTAT:


PROCESSESS: