Lots of pkgacct processes

[dchepishev]

Hi guys,

Can anyone help me with the following problem:
Since 2 weeks I am noticing the following weird behavior of at least 4 of our servers.
Some of them are upgraded from cPanel10 to cPanel11 but one of them with the same problem i directly installed with cPanel11

The problem is lots of processes left from backup and doing nothing:

Something similar to this:
13726 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct drembul /localbkp/backups/cpbackup/daily backup
13731 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct vensali /localbkp/backups/cpbackup/daily backup
13733 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct jokerbg /localbkp/backups/cpbackup/daily backup
13735 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct veskotr /localbkp/backups/cpbackup/daily backup
13737 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct theking /localbkp/backups/cpbackup/daily backup
13739 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct bulvis /localbkp/backups/cpbackup/daily backup
13746 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct plamcom /localbkp/backups/cpbackup/daily backup
13751 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct natalia /localbkp/backups/cpbackup/daily backup
13753 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct pudrier /localbkp/backups/cpbackup/daily backup
13760 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct lattant /localbkp/backups/cpbackup/daily backup
13762 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct ivokoy /localbkp/backups/cpbackup/daily backup
13765 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct argotre /localbkp/backups/cpbackup/daily backup
13767 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct gatakka /localbkp/backups/cpbackup/daily backup
13769 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct kandidat /localbkp/backups/cpbackup/daily backup
13771 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct loukano /localbkp/backups/cpbackup/daily backup
13773 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct nacbezb /localbkp/backups/cpbackup/daily backup
13776 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct perfectv /localbkp/backups/cpbackup/daily backup
13778 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct webdesig /localbkp/backups/cpbackup/daily backup
13780 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct bgpage /localbkp/backups/cpbackup/daily backup
13782 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct librasl /localbkp/backups/cpbackup/daily backup
13788 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct victori /localbkp/backups/cpbackup/daily backup
13794 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct luckyar /localbkp/backups/cpbackup/daily backup
13796 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct yoga37 /localbkp/backups/cpbackup/daily backup
13798 ? TNs 0:00 rm -rf /localbkp/backups/cpbackup/daily/expert
13800 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct expert /localbkp/backups/cpbackup/daily backup
13802 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct bestrx /localbkp/backups/cpbackup/daily backup
13804 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct no1pres /localbkp/backups/cpbackup/daily backup
13806 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct enkeled /localbkp/backups/cpbackup/daily backup
13808 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct mainsta /localbkp/backups/cpbackup/daily backup
13810 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct blfabg /localbkp/backups/cpbackup/daily backup
13812 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct podkrep /localbkp/backups/cpbackup/daily backup
13814 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct dichev /localbkp/backups/cpbackup/daily backup
13816 ? TNs 0:00 /usr/bin/perl /scripts/pkgacct barbora /localbkp/backups/cpbackup/daily backup

One one of the machines I have like 320 such hanging processes. If I kill them they appear on the next backup run.

cPanel support is not helping me so far. They are not giving me any reasonable answer. I think they dont even bother to investigate the case.
The killed the processes on one machine and that was all. The processes were killed by Level 2 support. On the next backup run they appeared again and no help and no answers from cPanel since then.

The machines are running:
perl 5.8.8
WHM 11.11.0 cPanel 11.15.0-R17659

[Infopro]

Whats this one?
rm -rf /localbkp/backups/cpbackup/daily/expert

[dchepishev]

Whats this one?
rm -rf /localbkp/backups/cpbackup/daily/expert

This probably is the rm job when the backup is finished. The copied directory is being removed.

[dchepishev]

I found what the problem is but dont know how to fix it:

The problem occurs in the following situation:

I have running backup process started by crond
The process is in suspended state (T) due to a high server load.

In the moment when I stop exim the backup process starts a cycle and cycles the
remaining backup accounts very fast and exits. However it looks like sometimes
the newly cycled accounts hang because they are set into state (T) and the main
backup process exits.

This behavior is probably due to some broken pipe or so because when stopping
exim the process sendmail to which the output is being piped is killed too.

Here is the killed process:

root 2492 0.0 0.0 9152 1936 ? Ss Jun23 0:13 crond
root 20699 0.0 0.0 10328 2504 ? S 01:00 0:00 \_ crond
root 20700 0.0 0.2 13840 11176 ? SNs 01:00 0:00 \_/scripts/cpbackup
root 8057 0.0 0.0 3168 776 ? SN 17:32 0:00 | \_/usr/local/cpanel/bin/cpuwatch 9.0 /scripts/pkgacc
root 8058 0.1 0.4 19760 16584 ? RNs 17:32 0:00 | \_pkgacct - savovai
mailnull 21046 0.0 0.0 14944 4136 ? S 01:00 0:00 \_/usr/sbin/sendmail -FCronDaemon -i -odi -oem -oi -t

When I stop exim the process with pid 21046 is killed too and the backup
process is going crazy.

[Bravo]

I'm having the same problem, there are over 600 such processes.

Is there any way to restart the backup and remove the hung processes? The server load is quite low (less then 0.5) but these processes linger on.

[cpanelkenneth]

Redirect /scripts/cpbackup output to /dev/null in the crontab entriy. E.g.:

0 0 * * * /scripts/cpbackup > /dev/null 2>&1

This prevents CRON from starting a sendmail process, which is the the root of the issue.