lfd alert for high server load. Who's the culprit?

[schwim]

Hi there everyone,

I got a notice that the server load spiked through lfd. The problem is that I don't see the culprit in the report. The server load is .15 right now, which is actually about normal for it. These numbers are way off the chart comparatively.

Any help would be appreciated:

Time: Mon May 21 14:40:18 2007
1 Min Load Avg: 18.32
5 Min Load Avg: 6.99
15 Min Load Avg: 2.66
Running/Total Processes: 2/201

Output from ps:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 1744 516 ? S 2006 0:35 init [3]
root 2 0.0 0.0 0 0 ? S 2006 2:33 [migration/0]
root 3 0.0 0.0 0 0 ? SN 2006 0:05 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S 2006 2:01 [migration/1]
root 5 0.0 0.0 0 0 ? SN 2006 0:07 [ksoftirqd/1]
root 6 0.0 0.0 0 0 ? S< 2006 0:00 [events/0]
root 7 0.0 0.0 0 0 ? S< 2006 0:01 [events/1]
root 8 0.0 0.0 0 0 ? S< 2006 0:00 [khelper]
root 9 0.0 0.0 0 0 ? S< 2006 0:00 [kthread]
root 16 0.0 0.0 0 0 ? S< 2006 25:15 \_ [kblockd/0]
root 17 0.0 0.0 0 0 ? S< 2006 0:17 \_ [kblockd/1]
root 18 0.0 0.0 0 0 ? S< 2006 0:00 \_ [kacpid]
root 227 0.0 0.0 0 0 ? S< 2006 0:00 \_ [khubd]
root 290 0.0 0.0 0 0 ? S< 2006 0:00 \_ [aio/0]
root 291 0.0 0.0 0 0 ? S< 2006 0:00 \_ [aio/1]
root 378 0.0 0.0 0 0 ? S< 2006 0:00 \_ [kseriod]
root 440 0.0 0.0 0 0 ? S< 2006 0:00 \_ [kpsmoused]
root 458 0.0 0.0 0 0 ? S< 2006 0:00 \_ [ata/0]
root 459 0.0 0.0 0 0 ? S< 2006 0:00 \_ [ata/1]
root 465 0.0 0.0 0 0 ? S< 2006 0:00 \_ [scsi_eh_0]
root 466 0.0 0.0 0 0 ? S< 2006 0:00 \_ [scsi_eh_1]
root 474 0.0 0.0 0 0 ? S< 2006 0:00 \_ [scsi_eh_2]
root 1245 0.0 0.0 0 0 ? S< 2006 0:00 \_ [kauditd]
root 26551 0.0 0.0 0 0 ? S 01:49 0:02 \_ [pdflush]
root 29669 0.0 0.0 0 0 ? S 14:40 0:00 \_ [pdflush]
root 289 0.0 0.0 0 0 ? S 2006 50:22 [kswapd0]
root 475 0.0 0.0 0 0 ? S 2006 0:00 [hpt_wt]
root 492 0.0 0.0 0 0 ? S 2006 87:02 [kjournald]
root 678 0.0 0.0 1644 332 ? S<s 2006 0:00 udevd -d
root 901 0.0 0.0 3140 76 ? Ss 2006 0:00 kmodule -d
root 1163 0.0 0.0 0 0 ? S 2006 0:00 [khpsbpkt]
root 1173 0.0 0.0 0 0 ? S 2006 0:00 [knodemgrd_0]
root 1385 0.0 0.0 0 0 ? S 2006 0:00 [kjournald]
root 1450 0.0 0.0 0 0 ? S< 2006 16:00 [loop0]
root 1451 0.0 0.0 0 0 ? S 2006 2:30 [kjournald]
root 1702 0.0 0.0 1612 552 ? Ss 2006 8:29 syslogd -m 0
root 1704 0.0 0.0 1564 376 ? Ss 2006 0:09 klogd -x
root 1784 0.0 0.0 1556 400 ? Ss 2006 0:00 /usr/sbin/acpid
root 1796 0.0 0.0 4400 796 ? Ss 2006 0:01 /usr/sbin/sshd
root 1804 0.0 0.0 2176 728 ? Ss 2006 0:00 xinetd -stayalive -pidfile /var/run/xinetd.pid
root 1812 0.0 0.0 4380 1072 ? S 2006 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --pid-file=/var/lib/mysql/server.myserver.com.pid
mysql 1842 1.0 5.4 581764 112284 ? Sl 2006 3643:47 \_ /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql --user=mysql --pid-file=/var/lib/mysql/server.myserver.com.pid --skip-external-locking --port=3306 --socket=/var/lib/mysql/mysql.sock
dbus 3046 0.0 0.0 2556 364 ? Ss 2006 0:00 dbus-daemon --system
root 3061 0.0 0.0 4212 764 ? Ss 2006 0:00 hald --retain-privileges
root 3070 0.0 0.0 2136 388 ? S 2006 0:00 \_ hald-addon-acpi
root 3097 0.0 0.0 33644 720 ? Sl 2006 0:09 /usr/bin/hptsvr
root 3115 0.0 0.2 11056 4864 ? Ss 2006 86:33 /usr/bin/perl -w /usr/bin/mrtg /etc/mrtg/mrtg.cfg
root 3119 0.0 0.0 1548 368 tty1 Ss+ 2006 0:00 /sbin/mingetty tty1
root 3120 0.0 0.0 1552 368 tty2 Ss+ 2006 0:00 /sbin/mingetty tty2
root 3121 0.0 0.0 1548 368 tty3 Ss+ 2006 0:00 /sbin/mingetty tty3
root 3122 0.0 0.0 1552 368 tty4 Ss+ 2006 0:00 /sbin/mingetty tty4
root 3123 0.0 0.0 1552 368 tty5 Ss+ 2006 0:00 /sbin/mingetty tty5
root 3124 0.0 0.0 1552 368 tty6 Ss+ 2006 0:00 /sbin/mingetty tty6
named 321 0.0 1.1 61572 24168 ? Ssl 2006 173:41 /usr/sbin/named -u named -t /var/named/chroot
root 31313 0.0 0.0 4560 936 ? Ss Feb16 0:12 crond
root 3436 0.0 0.3 13228 6828 ? Ss Feb17 1:04 /usr/local/apache/bin/httpd -DSSL
root 17408 0.0 0.1 6024 3764 ? S May20 0:00 \_ /usr/bin/perl /usr/local/cpanel/bin/leechprotect
nobody 29108 1.3 2.7 69756 57244 ? S 14:32 0:06 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29488 4.4 2.7 69796 56216 ? S 14:37 0:08 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29497 2.2 0.7 69732 16412 ? S 14:37 0:03 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29498 6.6 2.9 69716 60096 ? S 14:37 0:10 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29499 4.4 2.6 69760 55388 ? S 14:37 0:07 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29500 5.1 1.8 70712 38376 ? S 14:37 0:08 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29535 2.4 2.7 69436 55952 ? S 14:38 0:03 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29539 7.2 2.8 69748 58780 ? S 14:38 0:09 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29541 5.7 2.8 69748 57956 ? S 14:38 0:07 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29542 2.5 0.3 69456 7240 ? S 14:38 0:03 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29545 4.1 0.5 26092 12116 ? S 14:38 0:05 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29546 5.5 2.8 69760 57812 ? S 14:38 0:06 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29547 6.8 2.7 69464 56960 ? S 14:38 0:08 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29548 3.5 2.7 69744 55688 ? S 14:38 0:04 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29553 2.9 2.7 69444 56320 ? S 14:38 0:03 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29557 5.3 2.9 70652 60832 ? S 14:38 0:06 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29558 5.6 2.9 69744 60056 ? S 14:38 0:06 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29574 6.3 2.9 69752 59920 ? S 14:38 0:06 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29576 4.6 2.9 69456 59728 ? S 14:38 0:04 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29578 4.9 2.9 69460 59632 ? S 14:38 0:04 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29579 4.3 2.8 69452 59020 ? S 14:38 0:03 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29581 5.2 2.8 70644 57668 ? S 14:38 0:04 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29582 4.3 2.7 69460 57120 ? S 14:38 0:03 \_ /usr/local/apache/bin/httpd -DSSL

[schwim]

Continued:

nobody 29590 4.6 2.7 69452 57040 ? S 14:38 0:03 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29591 4.3 2.9 69456 59736 ? S 14:39 0:03 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29593 4.7 2.9 69452 59740 ? S 14:39 0:03 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29594 4.3 2.9 69448 59728 ? S 14:39 0:03 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29600 4.8 2.9 69436 59676 ? S 14:39 0:03 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29604 1.7 0.6 22788 13060 ? S 14:39 0:01 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29605 5.1 2.9 69436 59672 ? S 14:39 0:03 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29608 1.2 0.4 19076 9412 ? S 14:39 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29609 1.9 0.4 19376 9748 ? S 14:39 0:01 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29610 1.4 0.7 25388 15744 ? S 14:39 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29611 2.5 0.7 25584 15944 ? S 14:39 0:01 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29612 1.6 0.7 24208 14380 ? S 14:39 0:01 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29613 2.3 0.4 19076 9448 ? S 14:39 0:01 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29614 0.9 0.4 19444 9768 ? S 14:39 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29615 0.9 0.6 22344 12544 ? S 14:39 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29616 1.1 0.6 23828 14348 ? S 14:39 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29617 0.8 0.6 22852 13032 ? S 14:39 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29722 0.0 0.2 13420 5192 ? S 14:40 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29725 0.0 0.2 13228 4612 ? S 14:40 0:00 \_ /usr/local/apache/bin/httpd -DSSL
nobody 29726 0.0 0.2 13228 4612 ? S 14:40 0:00 \_ /usr/local/apache/bin/httpd -DSSL
mailnull 19145 0.0 0.0 7876 1776 ? Ss Mar02 6:00 /usr/sbin/exim -bd
mailnull 29635 0.0 0.1 8680 3088 ? S 14:39 0:00 \_ /usr/sbin/exim -bd
mailnull 19150 0.0 0.0 7872 1736 ? Ss Mar02 0:00 /usr/sbin/exim -C /etc/exim_outgoing.conf -q60m
mailnull 19160 0.0 0.0 7868 1732 ? Ss Mar02 0:00 /usr/sbin/exim -tls-on-connect -bd -oX 465
root 19218 0.0 0.0 3136 1064 ? S Mar02 0:08 antirelayd
root 19402 0.0 0.1 12024 2364 ? S Mar02 3:46 chkservd
root 7590 0.0 0.0 6440 1336 ? Ss Mar15 0:03 pure-ftpd (SERVER)
root 7592 0.0 0.0 5976 932 ? S Mar15 0:00 /usr/sbin/pure-authd -s /var/run/ftpd.sock -r /usr/sbin/pureauth
root 3928 0.0 0.0 1548 296 ? S Mar22 0:04 /usr/sbin/courierlogger -pid=/var/run/imapd.pid -start -name=imapd /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 143 /usr/lib/courier-imap/sbin/imaplogin /usr/lib/courier-imap/bin/imapd Maildir
root 3929 0.0 0.0 1648 512 ? S Mar22 0:07 \_ /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 143 /usr/lib/courier-imap/sbin/imaplogin /usr/lib/courier-imap/bin/imapd Maildir
root 3935 0.0 0.0 1552 296 ? S Mar22 0:00 /usr/sbin/courierlogger -pid=/var/run/imapd-ssl.pid -start -name=imapd-ssl /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 993 /usr/lib/courier-imap/bin/couriertls -server -tcpd /usr/lib/courier-imap/sbin/imaplogin /usr/lib/courier-imap/bin/imapd Maildir
root 3936 0.0 0.0 1652 516 ? S Mar22 0:00 \_ /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 993 /usr/lib/courier-imap/bin/couriertls -server -tcpd /usr/lib/courier-imap/sbin/imaplogin /usr/lib/courier-imap/bin/imapd Maildir
root 3941 0.0 0.0 1552 296 ? S Mar22 0:24 /usr/sbin/courierlogger -pid=/var/run/pop3d.pid -start -name=pop3d /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 110 /usr/lib/courier-imap/sbin/pop3login /usr/lib/courier-imap/bin/pop3d Maildir
root 3942 0.0 0.0 1652 520 ? S Mar22 0:28 \_ /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 110 /usr/lib/courier-imap/sbin/pop3login /usr/lib/courier-imap/bin/pop3d Maildir
root 3947 0.0 0.0 1552 296 ? S Mar22 0:00 /usr/sbin/courierlogger -pid=/var/run/pop3d-ssl.pid -start -name=pop3d-ssl /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 995 /usr/lib/courier-imap/bin/couriertls -server -tcpd /usr/lib/courier-imap/sbin/pop3login /usr/lib/courier-imap/bin/pop3d Maildir
root 3948 0.0 0.0 1648 516 ? S Mar22 0:00 \_ /usr/lib/courier-imap/libexec/couriertcpd -address=0 -maxprocs=40 -maxperip=30 -nodnslookup -noidentlookup 995 /usr/lib/courier-imap/bin/couriertls -server -tcpd /usr/lib/courier-imap/sbin/pop3login /usr/lib/courier-imap/bin/pop3d Maildir
root 3953 0.0 0.0 1548 292 ? S Mar22 0:00 /usr/sbin/courierlogger -pid=/var/spool/authdaemon/pid -facility=mail -start /usr/libexec/courier-authlib/authdaemond
root 3954 0.0 0.0 1864 636 ? S Mar22 0:00 \_ /usr/libexec/courier-authlib/authdaemond
root 3955 0.0 0.0 1864 400 ? S Mar22 0:05 \_ /usr/libexec/courier-authlib/authdaemond
root 15413 0.0 0.4 10212 8708 ? S May16 0:21 | \_ /etc/authlib/authProg
root 3956 0.0 0.0 1864 400 ? S Mar22 0:05 \_ /usr/libexec/courier-authlib/authdaemond
root 14660 0.0 0.4 10204 8636 ? S May16 0:20 | \_ /etc/authlib/authProg
root 3957 0.0 0.0 1864 400 ? S Mar22 0:05 \_ /usr/libexec/courier-authlib/authdaemond
root 22078 0.0 0.4 10208 8716 ? S Apr17 2:31 | \_ /etc/authlib/authProg
root 3958 0.0 0.0 1864 400 ? S Mar22 0:06 \_ /usr/libexec/courier-authlib/authdaemond
root 28513 0.0 0.4 10208 8716 ? S Apr20 2:19 | \_ /etc/authlib/authProg
root 3959 0.0 0.0 1864 400 ? S Mar22 0:06 \_ /usr/libexec/courier-authlib/authdaemond
root 15287 0.0 0.4 10204 8704 ? S May16 0:20 \_ /etc/authlib/authProg
root 789 0.0 0.1 7276 2512 ? S Apr26 0:01 cpbandwd
root 850 0.0 0.4 13884 9784 ? SN Apr26 5:28 cpanellogd - sleeping for logs
mailnull 864 0.0 0.2 8088 4592 ? S Apr26 0:02 eximstats
mailman 875 0.0 0.0 9904 1388 ? Ss Apr26 0:00 /usr/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/mailmanctl -s start
mailman 887 0.0 0.3 9748 6384 ? S Apr26 0:04 \_ /usr/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=ArchRunner:0:1 -s
mailman 888 0.0 0.3 9716 6396 ? S Apr26 0:05 \_ /usr/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=BounceRunner:0:1 -s
mailman 889 0.0 0.3 9692 6396 ? S Apr26 0:04 \_ /usr/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=CommandRunner:0:1 -s
mailman 890 0.0 0.3 9752 6392 ? S Apr26 0:04 \_ /usr/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s
mailman 891 0.0 0.3 9736 6420 ? S Apr26 0:04 \_ /usr/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=NewsRunner:0:1 -s
mailman 892 0.0 0.3 9716 6456 ? S Apr26 0:05 \_ /usr/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s
mailman 893 0.0 0.3 9724 6384 ? S Apr26 0:05 \_ /usr/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=VirginRunner:0:1 -s
mailman 894 0.0 0.3 9732 6384 ? S Apr26 0:00 \_ /usr/bin/python2.4 /usr/local/cpanel/3rdparty/mailman/bin/qrunner --runner=RetryRunner:0:1 -s
root 25731 0.0 0.1 9580 3772 ? S May08 0:35 lfd - sleeping
root 29729 0.0 0.1 9580 3468 ? S 14:40 0:00 \_ lfd - (child) checking load...
root 29730 0.0 0.0 4456 924 ? R 14:40 0:00 \_ /bin/ps axuf
mailnull 31347 0.0 0.3 22096 6624 ? Ss 02:36 0:00 MailScanner: starting child
mailnull 21498 0.0 1.6 46352 34612 ? S 13:03 0:03 \_ MailScanner: waiting for messages
mailnull 21556 0.0 1.8 46560 37248 ? S 13:04 0:03 \_ MailScanner: finishing batch
mailnull 22646 0.0 1.2 46252 26588 ? S 13:16 0:03 \_ MailScanner: waiting for messages
root 31415 0.0 0.1 17012 2992 ? S 02:36 0:01 cpsrvd - waiting for connections
root 29716 1.0 0.1 10708 3848 ? S 14:40 0:00 /usr/sbin/exim -C /etc/exim_outgoing.conf -Mc 1HqClu-0007gg-GO
mailnull 29723 0.0 0.1 10708 2088 ? S 14:40 0:00 \_ /usr/sbin/exim -C /etc/exim_outgoing.conf -Mc 1HqClu-0007gg-GO
mailnull 29720 0.7 1.7 47808 36556 ? S 14:40 0:00 MailWatch SQL

[schwim]

Continued:

Output from vmstat:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 5 348048 54140 5544 92284 0 0 0 1 1 1 6 2 91 1

Any help would be greatly appreciated.

thanks,
json

[salvatore333]

Me too:

| Top Process | 99.9 | lfd - checking directories |


Anyway of tweaking lfd to not use so many system resources or do I have to upgrade my server to handle it?

Thank you

[xufeng]

Me too:

| Top Process | 99.9 | lfd - checking directories |


Anyway of tweaking lfd to not use so many system resources or do I have to upgrade my server to handle it?

Thank you

Go to "Firewall Configuration",
Set following parameters accordingly to your needs.

SYSLOG = 0 if you would like to reduce IO operations of lfd
LF_HTACCESS = 0 enable this option to "1" only if you know you are suffering from attacks against password protected directories

LF_MODSEC = 0 to reduce high logging rate in the Apache error log, you might want to enable this option only if you know you are suffering from attacks against web scripts

LF_DIRWATCH = 3600 to reduce directory checking frequencies (suggest 3600sec or more or to disable it); specify <20 may cause resource issues...

LF_INTEGRITY = 0 to reduce IO load (similar to directory watch, it will use system resource in checking files)

After altering all these parameters, your server score could be lower, but you may get better resource control.