httpd-DSSL processes not closing, creating HUGE server loads

[jeroman8]

yepp, same here, always had those..

 

[yaax]

After many problems I had with server overload, I have wrote script to prevent this.
This script must run every few minutes - 2 or 3 minutes and he check current server load.
If server load is more then some value - like 13 or 15 or any other value, he stop httpd and probably otrher services like exim mysql and cpanel, and after this he check every 20 seconds if server load if low then 2 it will restart all services and exit.

#!/usr/bin/perl

my $date = `date`;
chomp $date;

open (LOG, ">>./load.log") || die "Failed to open file ::$!";

if (!( -f "/usr/local/apache/logs/httpd.pid" )) {
print LOG "$date - httpd not running, exiting. \n";
close (LOG);
exit;
}

my $html = `uptime`;
$html =~ s/\n+//g;

$ma=index($html,'e: ')+3;
$ma1=index($html,',',$ma);
$ld=substr($html,$ma,$ma1-$ma);

print LOG "$date - $ld\n";
$veryhigh=0;

if ($ld>=12) {
system("/usr/local/apache/bin/apachectl","stop");
system("/etc/init.d/cpanel","stop");
print LOG "$date - httpd stopped \n";

if ($ld>50) {
$veryhigh=1;
}

$ct=0;
while ($ld >= 2) {
sleep(20);
system("/usr/local/apache/bin/apachectl","stop");
system("/etc/init.d/cpanel","stop");
$html = `uptime`;
$html =~ s/\n+//g;
$ma=index($html,'e: ')+3;
$ma1=index($html,',',$ma);
$ld=substr($html,$ma,$ma1-$ma);
$date = `date`;
chomp $date;
print LOG "$date - after stop load=$ld \n";
}

if ($veryhigh>0) {
system("/scripts/restartsrv_mysql");
system("/scripts/restartsrv_exim");
$date = `date`;
chomp $date;
print LOG "$date - mysql,exim,cpanel restarted \n";
}

system("/scripts/restartsrv_httpd");
system("/etc/init.d/cpanel","restart");
$date = `date`;
chomp $date;
print LOG "$date - httpd restarted \n";
}

close (LOG);

exit;

 

[Radio_Head]

very good ! Something like prm , perhaps better !

 

[Radio_Head]

I have understood that your script get first uptime value (server load, $ld , in latest 1 minute), and store it on log file with date.

If $ld is over 12 it stops apache and cpanel

However I have not understand what does this part of code , please can you explain ?

$ct=0;
while ($ld >= 2) {
sleep(20);
system("/usr/local/apache/bin/apachectl","stop");
system("/etc/init.d/cpanel","stop");
$html = `uptime`;
$html =~ s/\n+//g;
$ma=index($html,'e: ')+3;
$ma1=index($html,',',$ma);
$ld=substr($html,$ma,$ma1-$ma);
$date = `date`;
chomp $date;
print LOG "$date - after stop load=$ld \n";
}

if ($veryhigh>0) {
system("/scripts/restartsrv_mysql");
system("/scripts/restartsrv_exim");
$date = `date`;
chomp $date;
print LOG "$date - mysql,exim,cpanel restarted \n";
}

system("/scripts/restartsrv_httpd");
system("/etc/init.d/cpanel","restart");
$date = `date`;
chomp $date;
print LOG "$date - httpd restarted \n";
}

Thank you

 

[yaax]

Then while $ld is bigger then 2, it stops apache and cpanel every 20 seconds and check again - whether load is bigger then 2 - and then exit from loop and restart all services - exim, mysql, apache and cpanel.

 

[Radio_Head]

Then while $ld is bigger then 2, it stops apache and cpanel every 20 seconds and check again - whether load is bigger then 2 - and then exit from loop and restart all services - exim, mysql, apache and cpanel.

OK clear ! Your idea could be a start point to add a lot of other checks

 

[yaax]

OK clear ! Your idea could be a start point to add a lot of other checks

I did few changes, cpanel should not be stopped more then one time, if it was stooped to many times it need to restart stunnel as well after.

if ($ld>=12) {
        system("/usr/local/apache/bin/apachectl","stop");
        system("/etc/init.d/cpanel","stop");
        print LOG "$date - httpd stopped \n";
        
        if ($ld>50) {
                $veryhigh=1;
        }

    $ct=0;
    while ($ld >= 2) {
                sleep(20);
                system("/usr/local/apache/bin/apachectl","stop");
                #system("/etc/init.d/cpanel","stop");
                $html = `uptime`;
                $html =~ s/\n+//g;
                $ma=index($html,'e: ')+3;
                $ma1=index($html,',',$ma);
                $ld=substr($html,$ma,$ma1-$ma);
                $date = `date`;
                chomp $date;
                print LOG "$date - after stop load=$ld \n";
   }

   if ($veryhigh>0) {
        system("/scripts/restartsrv_mysql");
        system("/scripts/restartsrv_exim");
        $date = `date`;
        chomp $date;
        print LOG "$date - mysql,exim,cpanel restarted \n";
   }

   system("/scripts/restartsrv_httpd");
   system("/etc/init.d/cpanel","restart");
   system("/usr/bin/stunnel-4.04local","/usr/local/cpanel/etc/stunnel/default/stunnel.conf");
   $date = `date`;
   chomp $date;
   print LOG "$date - httpd restarted \n";
}

close (LOG);

exit;

 

[Radio_Head]

I did few changes, cpanel should not be stopped more then one time, if it was stooped to many times it need to restart stunnel as well after.

if ($ld>=12) {
        system("/usr/local/apache/bin/apachectl","stop");
        system("/etc/init.d/cpanel","stop");
        print LOG "$date - httpd stopped \n";
        
        if ($ld>50) {
                $veryhigh=1;
        }

    $ct=0;
    while ($ld >= 2) {
                sleep(20);
                system("/usr/local/apache/bin/apachectl","stop");
                #system("/etc/init.d/cpanel","stop");
                $html = `uptime`;
                $html =~ s/\n+//g;
                $ma=index($html,'e: ')+3;
                $ma1=index($html,',',$ma);
                $ld=substr($html,$ma,$ma1-$ma);
                $date = `date`;
                chomp $date;
                print LOG "$date - after stop load=$ld \n";
   }

   if ($veryhigh>0) {
        system("/scripts/restartsrv_mysql");
        system("/scripts/restartsrv_exim");
        $date = `date`;
        chomp $date;
        print LOG "$date - mysql,exim,cpanel restarted \n";
   }

   system("/scripts/restartsrv_httpd");
   system("/etc/init.d/cpanel","restart");
   system("/usr/bin/stunnel-4.04local","/usr/local/cpanel/etc/stunnel/default/stunnel.conf");
   $date = `date`;
   chomp $date;
   print LOG "$date - httpd restarted \n";
}

close (LOG);

exit;

thank you 10000 !

 

[Radio_Head]

Some consideration ;

I think that your script toghter with uptime should check also
ps (grepping for example apache , exim and other things) ,
apache processes , and mysql processes.

Infact sometime teh scripts stops apache (also with few apache processes) while I have
/usr/bin/updatedb -f NFS,SMBFS,NCPFS,PROC,DEVPTS -e /tmp,/var/tmp,/usr/tm

Only some suggestion to make it more complex and useful.

 

[ispro]

What about "zombie" process?..

When MySQL got a very high load it cannot be restarted via no restart mysql service nor via /scripts/restartsrv_mysql

It's sometimes needed to execute killall -9 mysqld several times and then service mysql start

Can anyone comment this? Or point to the right direction to right restart script.

 

[daWeazy]

Seems this thread went from fixing the real problem to finding a temporary solution to a problem that will always exist unless someone really finds the correct solution. I have been fighting this problem for a Month.

 

[yaax]

Seems this thread went from fixing the real problem to finding a temporary solution to a problem that will always exist unless someone really finds the correct solution. I have been fighting this problem for a Month.

I think it is just problem of overloaded servers.
The real solution would be only upgrade of hardware or remove some too active sites from this server.

Thats why because I cannot do 2 above steps, I need this script...

 

[yaax]

I made few fixes in the script, like renice of this process and preventing this script from executing twice at same time:

#!/usr/bin/perl
system "/usr/bin/renice +20 -p $$ >/dev/null 2>&1";

my $date = `date`;
chomp $date;

open (LOG, ">>./load.log") || die "Failed to open file ::$!";

if (!( -f "/usr/local/apache/logs/httpd.pid" )) {
  print LOG "$date - httpd not running, exiting. \n";
  close (LOG);
  exit;
}

$pidfile="load.pid";
if (-e $pidfile)
{
  # aha, pid file is here, but process could be dead by now

  unless (open(PIDFILE,$pidfile)) {
    # too dangerous to start because I can't read old PID
    print LOG "$date - cannot read old pid, exiting. \n";
    close (LOG);
    exit 1;
  }
  my $oldpid=<PIDFILE>;
  close PIDFILE;

  # see if there is a process with such pid
  if ($oldpid > 1 && kill(0,$oldpid)) {
    # another proccess is running already
    print LOG "$date - script already running, exiting. \n";
    close (LOG);
    exit 1;
  } else {
    # that process is long dead
    print LOG "$date - found dead process $oldpid in PID file. \n";
  }
}

open (PID, ">$pidfile") or die;
print PID $$;
close(PID);

my $html = `uptime`;
$html =~ s/\n+//g;

$ma=index($html,'e: ')+3;
$ma1=index($html,',',$ma);
$ld=substr($html,$ma,$ma1-$ma);

print LOG "$date - $ld\n";
$veryhigh=0;

if ($ld>=12) {
        system("/usr/local/apache/bin/apachectl","stop");
        system("/etc/init.d/cpanel","stop");
        print LOG "$date - httpd stopped \n";
        #system("/root/top.cron");
        if ($ld>50) {
                $veryhigh=1;
        }

    $ct=0;
    $loadlevel=3;
    while ($ld >= $loadlevel) {
                sleep(20);
                system("/usr/local/apache/bin/apachectl","stop");
                #system("/etc/init.d/cpanel","stop");
                $html = `uptime`;
                $html =~ s/\n+//g;
                $ma=index($html,'e: ')+3;
                $ma1=index($html,',',$ma);
                $ld=substr($html,$ma,$ma1-$ma);
                $date = `date`;
                chomp $date;
                print LOG "$date - after stop load=$ld \n";
                $ct=$ct+1;
                if ($ct>50) {
                    $loadlevel=6;
                }
   }

#   system("/scripts/restartsrv_httpd");
#   $date = `date`;
#   chomp $date;
#   print LOG "$date - httpd restarted \n";

   if ($veryhigh>0) {
        system("/scripts/restartsrv_mysql");
        system("/scripts/restartsrv_exim");
        $date = `date`;
        chomp $date;
        print LOG "$date - mysql,exim,cpanel restarted \n";
   }

   system("/scripts/restartsrv_httpd");
   system("/etc/init.d/cpanel","restart");
   system("/usr/bin/stunnel-4.04local","/usr/local/cpanel/etc/stunnel/default/stunnel.conf");
   $date = `date`;
   chomp $date;
   print LOG "$date - httpd restarted \n";
}

close (LOG);

unlink $pidfile;

exit;

 

[yaax]

Some consideration ;

I think that your script toghter with uptime should check also
ps (grepping for example apache , exim and other things) ,
apache processes , and mysql processes.

Infact sometime teh scripts stops apache (also with few apache processes) while I have
/usr/bin/updatedb -f NFS,SMBFS,NCPFS,PROC,DEVPTS -e /tmp,/var/tmp,/usr/tm

Only some suggestion to make it more complex and useful.

updatedb process you should better disable completely by deleteing it from daily cron tab, or at least remove renice command from its script.
If I need to use locate command I am executing updatedb manually when I really need it.

I developed this script in order to prevent httpd overload mainly by apache processes, so if someone have other problems, he can make apropriate changes himself.

 

[The MAzTER]

i got the same problem as well
this is realy anoying.

anyone heard something from cpanel yet?

 

[famous]

Same here as well. anyone find a fix?? =)

Shane

 

[claudio]

unfortunatly i have to say same here too: (

Guys

after upgrading due to taht security issue stuff

overloads are appearing all the time in server status page

memory was high so i restarted and used some hints posted

here regarding to syslog.conf and it seems to be fine but CPU

usage is extremally high

i already checked and restarted mysql whose seems to be consuming

a lot and so...

please any clue...

Regards

Claudio

 

[XPerties]

I think it is just problem of overloaded servers.
The real solution would be only upgrade of hardware or remove some too active sites from this server.

Thats why because I cannot do 2 above steps, I need this script...

You are soooooooo wrong. My server has been stable for months and I have not added any new clients on it nor has any changes been made except a recent upgrade of cpanel. My loads have gone form .021 average to 39.+

All TOP shows is httpd with 391% of cpu usage.

 

[DrGreen]

You are soooooooo wrong. My server has been stable for months and I have not added any new clients on it nor has any changes been made except a recent upgrade of cpanel. My loads have gone form .021 average to 39.+

All TOP shows is httpd with 391% of cpu usage.

same here bro, i just got myself a brand new box, Xeon EM64T 2.8 GHz on Centos 3.3
Done transfering this night and the load went up to 70. and higher. I really need a solution for this

 

[yaax]

same here bro, i just got myself a brand new box, Xeon EM64T 2.8 GHz on Centos 3.3
Done transfering this night and the load went up to 70. and higher. I really need a solution for this

Different servers have different problems.

My server had problems with overloads with too active sites, so I solved it by utility scripts restarting services.

If new server server have such problems it could be many other reasons for high load....