php-fpm pool processes causing high load

[adeyjones]

Good morning

This morning I had a few complaints from customers that there websites were unresponsive, I logged in to WHM and the server load was 60+, when I looked at the list of processes there was 40-50 processes in the list from all accounts saying "php-fpm: pool sitename.com"

While looking at this, I was simultaneously working on another server and noticed the load on that one was around 30-40, looked at the process list on that one and that was also php-fpm pool processes. The first server I had to reboot to settle, the second settled on its own.

- Is this a process that runs on all servers at the same time?
- What is it? What does it do? and is it required?
- Why does it take up so much resource to the point that the sites on the server become unresponsive?
- How can I improve the performance of this?

Thank you.

 

[cPRex]

Hey there! PHP-FPM is the default PHP handler for all accounts - you can find more details on that here:

PHP-FPM | cPanel & WHM Documentation

docs.cpanel.net

The more likely explanation is that one account (or more) was using a lot of website resources at that time, leading to this issue, as the process itself runs like any other daemon and only uses resources when necessary.

 

[eservice]

I have this same problem maybe once a month. Usually the whole server becomes unresponsive and I have to give it a reboot. After a reboot there is a couple of minutes of time to login, "sudo iotop -o" to find out the problem user, and give it a penalty with "sudo chmod 0 /home/user1/public_html". Usually after couple of hour of penalty it is safe to change permission back to 750 and the problem has gone away.

What can this problem be? My guess is that It has to do something with php-fpm hammering MySQL database internally since network traffic is almost nonexistent.

How can this behavior be fixed?
How can I prevent one user from hogging all the I/O and crash the server?

 

[cPRex]

@eservice - it's impossible for us to say what the issue may be with just that information. If you're able to reproduce the issue with a specific account, feel free to submit a ticket to our team so we can do some additional investigation.

 

[eservice]

I am not able to reproduce this issue. It seems to be quite random and happens to one of 5 different accounts randomly maybe once or twice a month.

How can I investigate this further for you to share your thoughts? Or how can I prevent one user from hogging all the I/O and basically crash the server?

 

[cPRex]

@eservice - the best way to keep the users isolated is to use the CloudLinux operating system:

Introduction

|Features|CloudLinux OS Solo|CloudLinux OS Admin|CloudLinux OS Shared|CloudLinux OS Shared Pro| |:-:|:-:|:-:|:-:|:-:| |Maximum amount of hosting accounts|1|5|unlimited|unlimited| |Resource Limits (LVE)|No|Yes|Yes|Yes*| |Cage FS|Yes|Yes|Yes|Yes| |MySQL Governor|No|No|Yes|Yes| |PHP...

docs.cloudlinux.com

While there is an additional cost for this tool, it's ideal for shared hosting environments and is designed to resolve exactly the issues you're seeing.

 

[WorkinOnIt]

I also want to add my comment here to say the enabling php-fpm can lead to a high load situation.

This happens for me in a server with 5 accounts and only two are php-fpm enabled.

I notice the load increase, and then the only way to resolve it is to use Home / System Health / Process Manager to kill the users processes.

I understand that php-fpm can get "caught in a loop". Is there any way to limit the resources, without using cloud Linux?

 

[cPRex]

If you're using issues with PHP-FPM we'd like to see a ticket on that as it shouldn't happen.

 

[WorkinOnIt]

If you're using issues with PHP-FPM we'd like to see a ticket on that as it shouldn't happen.

OK I will think about a ticket, but for now have been playing around with the resources in PHP=FPM Settings under the user in MultiPHP Manager / domain settings.

On my test machine (4GB ram, 2CPU, 3 user accounts) I have the following:
Max Requests: 200
Max Children: 25
Process Timeout: 20

I reached these values using the calculation of :

4GB RAM for PHP-FPM processing with an average value of 20MB consumed by each child.

Maximum RAM:
PHP-FPM / RAM consumed by each child = Maximum children value for PHP-FPM

4000 / 20 = 200.

When I simulate 50 concurrent connections, the server locks up and load reaches 50 ! I can only recover by killing the user's processes in WHM: Home / System Health / Process Manager.

Is there any other way to reduce the bottlenecks?

 

[cPRex]

That just doesn't sound like normal behavior to me at all - we'd really like to see a ticket on that one.

 

[chezmohamed]

Hello I have the same problem with a 6VCORE 30GB server that happened to me today.
I have five accounts, and I’m sure it’s not from the customer’s charge.
I hope you will find a solution to this problem.
it is normal behaviour

 

[chezmohamed]

I think I found where the problem is coming from. after checking netstat on the server (in full Load Averages more than 30) I found an IP address trying to communicate with several domains at the same time so I blocked IP on the firewall and the overload problem is solved I hope this solution will help more people.
the IP address is 102.222.177.130

 

[cPRex]

@chezmohamed - I'm glad you were able to find that issue and fix it!