CURL problem, Curl is taking the server load to 40+ Help Please!

[badawi]

Hi,

I really hope you guys can give me a hand in this issue.
I have had this box hacked 3 days ago, we installed a new OS and fresh CPanel/WHM installation.
I recompiled Apache as well with CURL.
The problem is that the server is going in enormous loads (never before went over 2, now it is reaching 40s

The server is 3.2GH HT with 1GB RAM, Fedora i686

Here is the output of top:

top - 06:18:47 up 16:25, 1 user, load average: 30.45, 26.19, 21.20
Tasks: 229 total, 31 running, 196 sleeping, 2 stopped, 0 zombie
Cpu(s): 20.8% us, 76.7% sy, 0.0% ni, 0.0% id, 2.5% wa, 0.0% hi, 0.0% si
Mem: 967480k total, 844780k used, 122700k free, 101580k buffers
Swap: 2096440k total, 219192k used, 1877248k free, 83440k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6328 nobody 25 0 5744 1584 1372 R 7.0 0.2 35:17.34 curl
22173 nobody 25 0 5740 1652 1372 R 7.0 0.2 6:47.57 curl
22915 nobody 25 0 5736 1648 1372 R 7.0 0.2 5:51.47 curl
23987 nobody 25 0 5744 1656 1372 R 7.0 0.2 4:48.85 curl
29236 nobody 25 0 5744 1656 1372 R 7.0 0.2 0:54.90 curl
29796 nobody 25 0 5740 1652 1372 R 7.0 0.2 0:39.33 curl
31062 nobody 25 0 5740 1652 1372 R 7.0 0.2 0:03.93 curl
32398 nobody 25 0 5740 1556 1276 R 6.6 0.2 76:02.72 curl
11519 nobody 25 0 5744 1652 1372 R 6.6 0.2 26:19.65 curl
19538 nobody 25 0 5740 1652 1372 R 6.6 0.2 9:39.42 curl
20500 nobody 25 0 5740 1532 1372 R 6.6 0.2 8:10.44 curl
20591 nobody 25 0 5744 1656 1372 R 6.6 0.2 8:03.26 curl
20889 nobody 25 0 5740 1652 1372 R 6.6 0.2 7:42.45 curl
24390 nobody 25 0 5736 1648 1372 R 6.6 0.2 4:21.20 curl
29166 nobody 25 0 5744 1656 1372 R 6.6 0.2 0:59.12 curl
29347 nobody 25 0 5740 1652 1372 R 6.6 0.2 0:51.11 curl
29863 nobody 25 0 5740 1652 1372 R 6.6 0.2 0:39.38 curl
30044 nobody 25 0 5740 1652 1372 R 6.6 0.2 0:31.33 curl
30129 nobody 25 0 5744 1656 1372 R 6.6 0.2 0:27.50 curl
30212 nobody 25 0 5736 1648 1372 R 6.6 0.2 0:25.92 curl
30277 nobody 25 0 5736 1648 1372 R 6.6 0.2 0:24.82 curl
30422 nobody 25 0 5740 1652 1372 R 6.6 0.2 0:22.81 curl
30441 nobody 25 0 5740 1652 1372 R 6.6 0.2 0:21.61 curl
30790 nobody 25 0 5736 1648 1372 R 6.6 0.2 0:14.79 curl
30879 nobody 25 0 5740 1652 1372 R 6.6 0.2 0:12.91 curl
30965 nobody 25 0 5744 1656 1372 R 6.6 0.2 0:09.82 curl
21356 nobody 25 0 5744 1656 1372 R 6.3 0.2 7:18.64 curl
28412 nobody 25 0 5744 1652 1372 R 6.3 0.2 1:49.19 curl
29963 nobody 25 0 5740 1652 1372 R 6.3 0.2 0:35.60 curl
1458 nobody 15 0 28564 19m 5016 S 0.3 2.0 0:16.12 httpd
1461 nobody 15 0 31080 21m 5056 S 0.3 2.3 0:25.25 httpd


Look at the amount of CURL in there!


What can I do to fix it? if any information is required please ask and I will post it.

Regards and thanks to all who takes the time and effort to help.

[eth00]

Take a look at 'lsof -u nobody" to see what files nobody is accessing. It sounds like somebody is running some sort of a script that is causing it to run out of control. I would first try to stop apache then run it, with luck curl will keep running. If curl stops then I would temporarily block port 80 incoming. The problem is if you have a busy server apache will be accessing a lot at any given time.

[badawi]

Long live BurstNet, they fixed it for me.

Thanks for everyone that put the effort to read the post.
And sorry I don't know what Burst have done to fix it.

[correctsearch]

Yesterday I have installed CURL and I am having the same problem of server load.

I received an email as

The current load/uptime line
on the server at the time of
this email is
07:42:59 up 157 days, 21:30, 0 users, load average: 6.24, 6.25, 6.27
You should check the server to see why the load is so high and take steps to lower the load.

Could you please help me minimize my server load?

How to check what process is causing server load?

[Website Rob]

In shell run this command: ps aux

See what processes (on the far right) are being used by CURl and kill them.

[correctsearch]

I ran following command and have listed the output. I can see so many logs created by nobody. It is causing server load. What should I do stop server load

Command: ps aux

mysql 5142 0.0 1.9 36220 19680 ? S May15 0:00 /usr/sbin/m
root 10332 0.0 0.6 10176 7088 ? SN May15 0:11 cpanellogd
root 10334 0.0 0.4 8176 4772 ? S May15 0:00 cppop - acc
mailnull 10341 0.0 0.2 6684 2376 ? S May15 0:00 /usr/bin/pe
mysql 10342 0.0 1.9 36220 19680 ? S May15 0:00 /usr/sbin/m
mailman 10370 0.0 0.4 8600 4644 ? S May15 0:00 /usr/bin/py
nobody 10398 0.0 0.0 1732 612 ? S May15 0:00 /usr/local/
nobody 10400 0.0 0.1 4092 1096 ? S May15 0:00 entropychat
cpanel 10478 0.0 0.4 34420 4144 ? S May15 0:00 interchange
cpanel 10549 0.0 0.1 4652 1460 ? S May15 0:00 /usr/bin/st
root 10552 0.0 0.5 9548 5240 ? S May15 0:00 cpsrvd - wa
root 20012 15.8 0.0 6808 912 ? R May16 316:37 pico -w -z
compare 21682 0.0 0.6 10176 7088 ? SN May17 0:00 cpanellogd
compare 21683 0.0 0.0 1360 340 ? SN May17 0:00 /usr/local/
compare 21684 0.0 0.0 2328 16 ? TN May17 0:00 /usr/bin/pe
root 27070 0.0 0.5 12796 5296 ? S May17 0:00 /usr/local/
nobody 27076 0.0 0.8 14836 8376 ? S May17 0:10 /usr/local/
nobody 27100 0.0 0.8 14932 8320 ? S May17 0:07 /usr/local/
mailnull 27252 0.0 0.1 8044 1616 ? S May17 0:00 /usr/sbin/e
root 27276 0.0 1.9 22136 19712 ? S May17 0:00 /usr/bin/sp
root 27297 0.0 0.1 3392 1272 ? S May17 0:00 antirelayd
root 27324 0.0 2.1 24336 21972 ? S May17 0:08 spamd child
mysql 27345 0.0 1.9 36220 19680 ? S May17 0:02 /usr/sbin/m
nobody 27347 0.0 0.8 14856 8208 ? S May17 0:08 /usr/local/
nobody 27443 0.0 0.8 14840 8372 ? S May17 0:07 /usr/local/
mysql 16178 0.0 1.9 36220 19680 ? S 08:23 0:00 /usr/sbin/m

Command: lsof -u nobody

httpd 27443 nobody 64w REG 3,3 99956 10764290 /usr/local/apache/logs/ssl_engine_log
httpd 27443 nobody 65w REG 3,3 0 10764530 /usr/local/apache/logs/ssl_mutex.27067
httpd 27443 nobody 66w REG 3,3 75822643 10764409 /usr/local/apache/logs/access_log
httpd 27443 nobody 67w REG 3,3 13164 6537621 /usr/local/apache/domlogs/mail.correctsearch.com
httpd 27443 nobody 68w REG 3,3 1296 6537622 /usr/local/apache/domlogs/2lookup.com
httpd 27443 nobody 69w REG 3,3 1242 6537617 /usr/local/apache/domlogs/al-nawadir.net
httpd 27443 nobody 70w REG 3,3 0 6537469 /usr/local/apache/domlogs/guugul.com
httpd 27443 nobody 71w REG 3,3 15387 6537567 /usr/local/apache/domlogs/detroitguide.net
httpd 27443 nobody 72w REG 3,3 162189 6537413 /usr/local/apache/domlogs/correctsearch.net
httpd 27443 nobody 73w REG 3,3 168035 6537471 /usr/local/apache/domlogs/sitetoregister.com
httpd 27443 nobody 74w REG 3,3 623117 6537599 /usr/local/apache/domlogs/explorewww.com
httpd 27443 nobody 75w REG 3,3 45563599 6537401 /usr/local/apache/domlogs/correctsearch.com
httpd 27443 nobody 76w REG 3,3 104962 6537537 /usr/local/apache/domlogs/artbyagar.com

Please give me suggestion, how to take control of this.