CentOS/WHM hangs for a few minutes

[adriani]

Hello,

I've recently purchased a server with CentOS 5.7 and WHM 11.30.5.
I've configured 2 websites on the server, which work great.
Now the issue occurs to some visitors when they try to access the websites. The server stops responding for them for a few minutes, and then it comes back. During the time the server stops responding for some visitors, it is responding to others. So basically, the server is not down.
Is this a security setting somewhere ? Did this happened to somebody ?

Many thanks.

 

[ruzbehraja]

Now the issue occurs to some visitors when they try to access the websites. The server stops responding for them for a few minutes, and then it comes back. During the time the server stops responding for some visitors, it is responding to others.

• How often does this happen?

• When the server stops responding, is WHM / SSH accessible?

Can you paste the output of the top -c command here?
For that you need to go to SSH and type:

top -c

Is this a security setting somewhere ? Did this happened to somebody ?

This could be an Apache (Max Clients) setting, but for that we would need to know what type of websites you are running and how many visitors they are serving.

 

[adriani]

Hello ruzbehraja,

It happens a few times a day.

When it stops, neither WHM / SSH is not accessible.

Content of 'top -c':

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 15 0 2160 420 392 S 0.0 0.0 0:02.65 init [3]
2 root RT -5 0 0 0 S 0.0 0.0 0:00.16 [migration/0]
3 root 34 19 0 0 0 S 0.0 0.0 0:00.05 [ksoftirqd/0]
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/0]
5 root RT -5 0 0 0 S 0.0 0.0 0:00.10 [migration/1]
6 root 34 19 0 0 0 S 0.0 0.0 0:00.04 [ksoftirqd/1]
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/1]
8 root RT -5 0 0 0 S 0.0 0.0 0:00.11 [migration/2]
9 root 34 19 0 0 0 S 0.0 0.0 0:00.00 [ksoftirqd/2]
10 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/2]
11 root RT -5 0 0 0 S 0.0 0.0 0:00.07 [migration/3]
12 root 39 19 0 0 0 S 0.0 0.0 0:00.02 [ksoftirqd/3]
13 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/3]
14 root RT -5 0 0 0 S 0.0 0.0 0:00.14 [migration/4]
15 root 34 19 0 0 0 S 0.0 0.0 0:00.00 [ksoftirqd/4]
16 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/4]
17 root RT -5 0 0 0 S 0.0 0.0 0:00.10 [migration/5]
18 root 34 19 0 0 0 S 0.0 0.0 0:00.01 [ksoftirqd/5]
19 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/5]
20 root RT -5 0 0 0 S 0.0 0.0 0:00.09 [migration/6]
21 root 39 19 0 0 0 S 0.0 0.0 0:00.00 [ksoftirqd/6]
22 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/6]
23 root RT -5 0 0 0 S 0.0 0.0 0:00.07 [migration/7]
24 root 34 19 0 0 0 S 0.0 0.0 0:00.01 [ksoftirqd/7]
25 root RT -5 0 0 0 S 0.0 0.0 0:00.00 [watchdog/7]
26 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [events/0]
27 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [events/1]
28 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [events/2]
29 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [events/3]
30 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [events/4]
31 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [events/5]
32 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [events/6]
33 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [events/7]
34 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 [khelper]
35 root 19 -5 0 0 0 S 0.0 0.0 0:00.01 [kthread]
45 root 10 -5 0 0 0 S 0.0 0.0 0:00.03 [kblockd/0]
46 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 [kblockd/1]
47 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [kblockd/2]
48 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [kblockd/3]
49 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [kblockd/4]
50 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [kblockd/5]
51 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 [kblockd/6]
52 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 [kblockd/7]
53 root 16 -5 0 0 0 S 0.0 0.0 0:00.00 [kacpid]


I don't think it has to do with the number of maximum users, because one of the sites it preety new and not so known. We had the same problem with this site when we first configured it.

Can't it be a firewall config or something ?

Many thanks!

 

[storminternet]

There are some considerations for server hangs all of sudden.

1)If your application starts consuming server resources. This may includes your databases, php scripts, crons, etc..
Check server processes with the top -c command, ps -aux and find out the resource intensive application.

Additionally you can use mysqladmin pr to view resource consuming database.

2)If connection on apache goes on increasing all of sudden.
Use of netstat will show you total connections on port 80.

Simple command to check no. of connection on port 80.

netstat -anp | grep :80 | wc -l

You will find many more options if you google for it.

3)If server hardware is faulty. You need to check RAM performance, Disk I/O usage, cpu speed etc..

4)Incompatible kernel version

For option 3 and 4 You need to take the help of your host or dc engineers to check server performance.

 

[cPanelTristan]

You can check if it is MaxClients in httpd.conf by running the following:

grep -i maxclient /usr/local/apache/logs/error_log

When it is happening, you should also review the error_log for the site(s) in question to see what is happening. Most likely, some type of error would be logging.

I would also shut down any firewall temporarily (disable the firewall or flush iptables) to see if this is firewall-related.

Thanks!

 

[adriani]

Hello all,

First of all, thank you for replying to my problem.

Secondly, I believe this is a firewall issue.
I've asked the support team from the hosting company to check it again, and they said they reinstalled the firewall and tweaked the settings.

Basically, this is why I believe it's a firewall issue:

1 - We have another server, a development server, running on a Ubuntu with WebMin. A few sites from our dev server is connecting to the mysql server from the live server, the one with CentOs/WHM issues.
Today, the sites from the dev server could not connect to the mysql server: 'Unable to connect to ......'
I added the dev server's IP in csf.allow and the problem was fixed.

2 - Before adding any IPs to csf.allow, the problem used to happen in our office too. The way we fixed it was by running: service iptables restart - recommended by the support team

3 - The number of connections on port 80 was between 20 and 1200 - BUT the sites worked great for the computers in the office

4 - We added a firefox plugin for proxies, and although the number of connections on port 80 was high, the sites worked. When they didn't worked, we ran the service iptables restart command, and then they worked.

5 - the command grep -i maxclient /usr/local/apache/logs/error_log did not displayed anything

6 - I have checked the CPU usage when the sites stoped working, and no problems there

7 - I disabled firewall for a while - and there were no problems at all. I enabled it back.

8 - I configured the firewall settings to 'LOW' - most of the settings were set to 'LOW', except for a few, as it can be seen from the attached print screen. Now it seems to be working ok.



As a conclusion, I would say it's a firewall problem, but I am no expert. Is it ok though if the settings are set to 'LOW' ?

Many thanks again!

 

[ruzbehraja]

During the time the server stops responding for some visitors, it is responding to others. So basically, the server is not down.

What are the websites serving up? Are there any MySQL queries? Are the users doing something specific to get the error?


Can you post the Apache settings which you are currently running with?


You may want to install the system snapshot utility as explained in this post:
https://forums.cpanel.net/f5/server-goes-down-lot-lately-206072.html#post848931

 

[cPanelTristan]

Hello,

Since you are using CSF, it sounds like LFD was likely blocking users for some reason and that's why other users could still connect. When it was blocking users, you could view the logs at /var/log/lfd.log to see why it blocked someone.

I would like to point out that CSF is an unrelated product from another company (ConfigServer Security) that has a forum at this location:

ConfigServer Scripts Forum

Going forward, if you experience any further issues with that product, I would suggest posting there. For example, we wouldn't know the implications of changing to LOW level on their software, since it isn't our software. They would know, though, on their forum.

Glad you got it tracked down to the firewall.

Thanks!