Unwanted second httpd server instances

[eutelnet]

I noticed weird behavior with apache on one server.

I'm running EA3 (mpm event) on Cloudlinux 7.
Everything was until couple days ago when I noticed (usually discovered by increase in apache load) there are where to much instances running

$ps -aux | grep httpd

root      807675  0.1  0.7 225368 126288 ?       Ss   14:40   0:00 /usr/local/apache/bin/httpd -k graceful
nobody    807687  0.6  0.8 2150596 145028 ?      Sl   14:40   0:03 /usr/local/apache/bin/httpd -k graceful
nobody    807688  0.4  0.9 2150996 146008 ?      Sl   14:40   0:02 /usr/local/apache/bin/httpd -k graceful
nobody    807736  1.3  1.0 2150996 163476 ?      Sl   14:40   0:08 /usr/local/apache/bin/httpd -k graceful
nobody    807788  3.3  0.9 2151596 156272 ?      Sl   14:40   0:20 /usr/local/apache/bin/httpd -k graceful
root      808351  0.1  0.7 225372 126292 ?       Ss   14:41   0:00 /usr/local/apache/bin/httpd -k start
nobody    808362  0.3  0.8 2151000 141212 ?      Sl   14:41   0:01 /usr/local/apache/bin/httpd -k start
nobody    808375  1.7  0.9 2151200 148360 ?      Sl   14:41   0:10 /usr/local/apache/bin/httpd -k start
nobody    808410  1.0  0.9 2151000 150828 ?      Sl   14:41   0:06 /usr/local/apache/bin/httpd -k start
nobody    808455  2.0  1.0 2152000 168388 ?      Sl   14:41   0:11 /usr/local/apache/bin/httpd -k start
nobody    810368 56.3  0.9 2150596 146092 ?      R    14:47   1:34 /usr/local/apache/bin/httpd -k graceful
nobody    810380 36.9  0.9 2150596 146156 ?      R    14:47   0:59 /usr/local/apache/bin/httpd -k graceful
nobody    810427 24.9  0.9 2150596 147148 ?      R    14:48   0:37 /usr/local/apache/bin/httpd -k graceful
nobody    810432 19.1  0.9 2150596 147172 ?      R    14:48   0:28 /usr/local/apache/bin/httpd -k graceful
nobody    810442 20.4  0.9 2150596 147072 ?      R    14:48   0:30 /usr/local/apache/bin/httpd -k graceful
nobody    810462 39.1  0.9 2150596 147504 ?      R    14:48   0:55 /usr/local/apache/bin/httpd -k graceful
nobody    810469 66.0  0.9 2150596 147884 ?      R    14:48   1:32 /usr/local/apache/bin/httpd -k graceful
nobody    810473 19.4  0.9 2150596 148036 ?      R    14:48   0:26 /usr/local/apache/bin/httpd -k graceful
nobody    810479 25.7  0.9 2150596 148640 ?      R    14:48   0:35 /usr/local/apache/bin/httpd -k graceful
nobody    810480 16.8  0.9 2150596 148492 ?      R    14:48   0:22 /usr/local/apache/bin/httpd -k graceful
nobody    810501 66.7  0.9 2150596 149392 ?      R    14:48   1:26 /usr/local/apache/bin/httpd -k graceful
nobody    810586 50.2  0.9 2150596 149716 ?      R    14:48   0:51 /usr/local/apache/bin/httpd -k graceful
nobody    810605 31.7  0.9 2150596 149824 ?      R    14:48   0:31 /usr/local/apache/bin/httpd -k graceful
nobody    810616 42.1  0.9 2150596 149928 ?      R    14:48   0:39 /usr/local/apache/bin/httpd -k graceful
nobody    810619 68.3  0.9 2150596 149916 ?      R    14:49   1:03 /usr/local/apache/bin/httpd -k graceful
nobody    810651 69.1  0.9 2150596 150404 ?      R    14:49   1:01 /usr/local/apache/bin/httpd -k graceful
nobody    810666 44.9  0.9 2150596 150740 ?      R    14:49   0:35 /usr/local/apache/bin/httpd -k graceful
nobody    810671 52.8  0.9 2150596 150740 ?      R    14:49   0:41 /usr/local/apache/bin/httpd -k graceful
nobody    810685 39.2  0.9 2150596 150744 ?      R    14:49   0:28 /usr/local/apache/bin/httpd -k graceful
nobody    810705 46.8  0.9 2150596 150800 ?      R    14:49   0:30 /usr/local/apache/bin/httpd -k graceful
nobody    810716 26.5  0.9 2150596 151048 ?      R    14:49   0:16 /usr/local/apache/bin/httpd -k graceful
nobody    810726 11.8  0.9 2150596 151180 ?      R    14:49   0:07 /usr/local/apache/bin/httpd -k graceful
nobody    810728 11.8  0.9 2150596 151180 ?      R    14:49   0:06 /usr/local/apache/bin/httpd -k graceful
nobody    810740 24.6  0.9 2150596 151244 ?      R    14:49   0:13 /usr/local/apache/bin/httpd -k graceful
nobody    810742 55.7  0.9 2150596 151556 ?      R    14:49   0:27 /usr/local/apache/bin/httpd -k graceful
nobody    810744 10.9  0.9 2150596 151544 ?      R    14:49   0:05 /usr/local/apache/bin/httpd -k graceful
nobody    810753 24.8  0.9 2150596 151852 ?      R    14:49   0:11 /usr/local/apache/bin/httpd -k graceful
nobody    810754 10.9  0.9 2150596 152200 ?      R    14:49   0:04 /usr/local/apache/bin/httpd -k graceful
nobody    810791 23.5  0.9 2150596 152204 ?      R    14:49   0:09 /usr/local/apache/bin/httpd -k graceful
nobody    810800 48.1  0.9 2150596 152572 ?      R    14:49   0:18 /usr/local/apache/bin/httpd -k graceful
nobody    811049 35.3  0.9 2150996 155104 ?      R    14:50   0:04 /usr/local/apache/bin/httpd -k graceful
nobody    811062 19.3  0.9 2155120 155132 ?      R    14:50   0:01 /usr/local/apache/bin/httpd -k graceful
nobody    811064 66.1  0.9 2150996 155280 ?      R    14:50   0:04 /usr/local/apache/bin/httpd -k graceful
nobody    811068 22.2  0.9 2150996 155280 ?      R    14:50   0:00 /usr/local/apache/bin/httpd -k graceful
nobody    811069 49.6  0.9 2150996 155428 ?      R    14:50   0:01 /usr/local/apache/bin/httpd -k graceful

After killall on httpd and server restart everything gets back to normal
$killall httpd && /scripts/restartsrv_httpd

root      812455  4.6  0.7 225372 126304 ?       Ss   14:52   0:00 /usr/local/apache/bin/httpd -k start
nobody    812460  1.5  0.8 1364168 131860 ?      Sl   14:52   0:00 /usr/local/apache/bin/httpd -k start
nobody    812462  2.7  0.8 1888456 136124 ?      Sl   14:52   0:00 /usr/local/apache/bin/httpd -k start
nobody    812509  0.3  0.7 774344 123912 ?       Sl   14:52   0:00 /usr/local/apache/bin/httpd -k start
nobody    812553  1.3  0.8 1626312 132000 ?      Sl   14:52   0:00 /usr/local/apache/bin/httpd -k start
root      812610  0.0  0.0 112652   952 pts/0    S+   14:52   0:00 grep httpd

I'm very positive this is not some temporary issue since it keeps getting back all over again for last 48hours.

Thank you,
Nemanja

Also, it seems that new httpd processes are sometimes spawn by httpd processes owned by nobody

 nobody    812462  6.8  0.9 2150600 158100 ?      Sl   14:52   0:27  \_ /usr/local/apache/bin/httpd -k start
nobody    813909 66.7  0.8 2150600 144136 ?      R    14:57   1:34  |   \_ /usr/local/apache/bin/httpd -k start
nobody    813930 41.3  0.8 2150600 144548 ?      R    14:57   0:57  |   \_ /usr/local/apache/bin/httpd -k start
nobody    813931 44.9  0.8 2150612 144748 ?      R    14:57   1:02  |   \_ /usr/local/apache/bin/httpd -k start
nobody    813937 46.8  0.8 2150600 145348 ?      R    14:57   1:04  |   \_ /usr/local/apache/bin/httpd -k start
nobody    813939 45.7  0.8 2150620 145620 ?      R    14:57   1:03  |   \_ /usr/local/apache/bin/httpd -k start
nobody    813987 40.6  0.9 2150600 146732 ?      R    14:57   0:47  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814191 25.0  0.9 2150600 147420 ?      R    14:58   0:19  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814235 28.5  0.9 2150600 147504 ?      R    14:58   0:19  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814239 73.0  0.9 2150600 147792 ?      R    14:58   0:48  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814240 63.6  0.9 2150600 147792 ?      R    14:58   0:41  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814249 21.3  0.9 2150600 147876 ?      R    14:58   0:12  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814261 22.7  0.9 2150600 147944 ?      R    14:58   0:12  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814268 58.9  0.9 2150600 148220 ?      R    14:58   0:29  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814271 93.8  0.9 2150600 148220 ?      R    14:58   0:46  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814278 22.7  0.9 2150600 148264 ?      R    14:58   0:11  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814283 22.9  0.9 2150600 148308 ?      R    14:58   0:11  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814285 27.2  0.9 2150600 148636 ?      R    14:58   0:13  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814286 26.0  0.9 2150600 148964 ?      R    14:58   0:12  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814287 55.8  0.9 2150600 148972 ?      R    14:58   0:26  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814299 58.0  0.9 2150600 148992 ?      R    14:58   0:24  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814328 20.8  0.9 2150600 149044 ?      R    14:58   0:07  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814333 93.8  0.9 2150600 149196 ?      R    14:58   0:32  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814346 60.8  0.9 2150600 149588 ?      R    14:58   0:17  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814392 26.0  0.9 2150600 149576 ?      R    14:59   0:04  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814410 99.6  0.9 2150600 151012 ?      R    14:59   0:12  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814414 51.6  0.9 2150600 151068 ?      R    14:59   0:06  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814430 95.7  0.9 2150600 151072 ?      R    14:59   0:03  |   \_ /usr/local/apache/bin/httpd -k start
nobody    814431 14.0  0.9 2150600 151124 ?      R    14:59   0:00  |   \_ /usr/local/apache/bin/httpd -k star

 

[cPanelMichael]

Hello,

It's normal for Apache to run as the "nobody" user. It's also normal to see multiple httpd processes running on the system. The following thread is a good place to start when troubleshooting load issues:

Troubleshooting high server loads on Linux servers

Thank you.