Apache hangs; problem with logrotate?

[GoWilkes]

Ever since I rebuilt Apache last month (it wasn't recognizing newly parked domains), I've been having a problem. Every few hours, Apache will just "hang", for anywhere from 1 to 15 minutes. I can SSH in, and I can log in to WHM, but the website itself is unresponsive.

When I log in to SSH and command "top -c", it will show that pretty much nothing is running, even though it's during peak hours. My Munin report shows a "dead" area in Apache Processes for that period, too.

After quite a bit of going back and forth with my server provider (it's a semi-managed server, not shared), they thought that the problem was that logrotate was taking too long, causing Apache to hang up.

At the time, I had it set to 50MB under Tweak Settings, but moved it up to 300MB in the hopes of minimizing the number of times it would rotate. That helped for a few days, then it started hanging again, so I moved it down to 10MB in the hopes that a smaller log would make it finish faster. But still, it's hanging on me several times a day.

It's currently still set to 10MB.

Being rather amateur when it comes to networking, I'm not entirely sure what files are supposed to be rotated. But when I go to /usr/local/apache/logs, my access log is 140MB. Under /usr/local/apache/domlogs/, the main log for my account is 537MB, and a second log for a subdomain is 62MB. Then, under /usr/local/apache/logs/archive/ (where I thought that logs were supposed to rotate), I have nothing but README.archive.

Does this means that logrotate isn't actually moving anything?

If it matters, I'm using CSF for the firewall. I've tried forcing an update of cPanel (the regular setting is RELEASE Builds (Automatic)), and rebuilding Apache, with no impact. Also, I'm running WHM 11.28.86, on a Redhat Enterprise 4 i686 server.

TIA,

Jason

 

[GoWilkes]

As an update, this file does not exist:

/etc/logrotate.d/apache/ (or /etc/logrotate.d/httpd)

I found this sample:

/usr/local/apache/logs/*_log {
notifempty
daily
rotate 1
missingok
compress
postrotate
/bin/kill -HUP `cat /usr/local/apache/logs/httpd.pid 2>/dev/null` 2> /dev/null || true
endscript
}

Should I create this file manually?

 

[cPanelTristan]

From what I am seeing on another cPanel machine, /etc/logrotate.d/httpd file should exist with the following content:

/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

You might want to try creating /etc/logrotate.d/httpd file manually with the above content.

 

[GoWilkes]

I started to create that, but before doing so, I should mention that I don't have the directory referenced in the first line:

/var/log/httpd/

I'm not entirely sure what this line does, but if it's supposed to point to my log files, the only path I can find is /usr/local/apache/logs/. Should I change the first line?

Two other notes:

1. I found today that the server DOES rotate suexec_log at 10MB. So, apparently, it's working, but it's not rotating my access_log or error_log.

2. I checked my second server, and it's rotating everything correctly, but doesn't have an /etc/logrotate.d/httpd file. So I'm guessing that this file isn't essential, after all? Unless maybe it's in another location. I dunno. The settings on these two servers seem the same, so I don't know why one's rotating logs and the other isn't.

 

[cPanelTristan]

Does the server that is rotating the logs have those logs checked in WHM > cPanel Log Rotation Configuration area?

 

[GoWilkes]

Yes; everything is selected except safeapacherestart_log and setupdbmap_log (which I assume is the default setting, because I don't remember changing anything here).

 

[cPanelTristan]

So both machines have those checked or just the one where it is working has those logs checked? If the non-working one doesn't have those logs checked, please try checking them on it. If it does, yes, please try changing the path to the existing path for the log files at /usr/local/apache/logs/*log instead.

 

[GoWilkes]

I had actually just checked the one that's NOT rotating, but I just now checked the other one. They're both identical on this setting.

I'll add the httpd file tonight; I'm about to start peak hours, and don't want to risk any more down time than I'm already getting

 

[GoWilkes]

FYI, I added the httpd file about an hour ago, but nothing has rotated yet. The access_log is currently just under 142MB, and WHM has the log threshold set to 10MB.

I'm not sure how often it checks the size of the logs to determine whether to rotate?

 

[cPanelTristan]

What is your configuration file set to show at /etc/logrotate.conf location?

 

[GoWilkes]

I hadn't seen this file before, so I'm assuming these are the default settings:

# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
monthly
create 0664 root utmp
rotate 1
}

 

[GoWilkes]

Still nothing.

I changed logrotate.conf to "daily" instead of "weekly" on 3/18/2011, and then in WHM, changed the threshold to 142MB. The access_log was over 141MB, and I figured that with it set to 10MB, there might be too much of a difference.

But, access_log went over 142MB today, and it still hasn't rotated.

I thought that I might add this to logrotate.conf (after the "/var/log/wtmp" script):

/usr/local/apache/logs/access.log {
rotate 5
size=10M
postrotate
/sbin/killall -HUP httpd
endscript
}


Think it might help?

 

[cPanelTristan]

You can try this. If it doesn't work, then please go ahead and open up a ticket, since it seems the cPanel logrotation option in WHM isn't really working properly either.

 

[GoWilkes]

Just curious, how do I manually force logrotate via SSH? Is it just:

logrotate -f