apache master/custom/global access log

SergeiF

Registered
Jun 29, 2014
2
1
1
cPanel Access Level
Root Administrator
Recently I had to rebuild a WHM server due to filesystem corruption. I was able to recover most of the functionality/customisation (the cpanel backup is bloody useless btw, will never ever rely on it again).
The version is WHM 11.44.0 (build 18).

One problem is I cannot replicate one customisation (even though it works on other WHM machines that we run).
The customisation in question is combined apache access log.

Right now it is real pain to monitor abuse on the machine, due to no central/master apache log. I cannot tail domlogs due to there are thousands of accounts on each machine.

The way I was able to have combined log is by creating/editing /var/cpanel/templates/apache2/main.local

and adding the following:

Code:
LogFormat "%V %a %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"" vcommon
CustomLog /var/log/httpd/vhost-access_log vcommon
just after this:
Code:
    <IfModule [% main.mainifmodulelogconfigmoduleifmodulelogiomodule.item.ifmodule %]>
[% FOREACH dir IN main.ifmodulelogconfigmodule.ifmodulelogiomodule.logformat.items -%]
        LogFormat [% dir.logformat %]
[% END -%]

    </IfModule>

</IfModule>
and before this:
Code:
<IfModule [% main.mainifmodulealiasmodule.item.ifmodule %]>
[% FOREACH dir IN main.ifmodulealiasmodule.scriptalias.items -%]
    ScriptAlias [% dir.url %] [% dir.path %]
[% END -%]

</IfModule>
Basically I have replicated what we have on other multiple machines (the machine that the combined log works is running WHM 11.38.2 (build 23) ).
It does not work on new machine. The only entires that are being logged are those that are not in virtual hosts....

Basically what I am after is a central/master/combined apache access log, so I can tail it and see which domain get what requests from which ip.

I have searched this forum for this, and documentation, there is a lot of misleading information, most of the forum posts are simply unanswered or answered incorrectly (eg I am not after /usr/local/apache/domlogs/, nor I am after /usr/local/apache/logs/access_log). I don't understand how people get away without having one? Do they not care about abuse? This log is a must when running shared hosting. It should be on by default.

So the question is: what is the correct way of setting up the combined apache log for all vhosts?
 

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
Right now it is real pain to monitor abuse on the machine, due to no central/master apache log. I cannot tail domlogs due to there are thousands of accounts on each machine.
Hello :)

To clarify, is the goal here to have one log file with the data from all of your Apache domlogs combined? If so, you could develop a bash script that combines all of the Apache domlogs into one single log file after the initial logging. Feel free to clarify or let us know if that's not what you are after.

Thank you.
 

SergeiF

Registered
Jun 29, 2014
2
1
1
cPanel Access Level
Root Administrator
Hi, The idea is to have a "live" log of all the domains combined. The bash script will not do, as it is much more efficient to tail the log (and perhaps grep for stuff of interest). Sometimes there is no time before server runs out of memory due to DoS.

In anyway, while my message was being aproved by moderator I figured out the answer:

The correct place to put the log line is in /var/cpanel/templates/apache2/vhost.local (cp from vhost.default):

Code:
...........

LogFormat "%V %a %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"" vcommon
CustomLog /var/log/httpd/vhost-access_log vcommon

</VirtualHost>
It can be placed right before closing </VirtualHost> tag in the file. The ssl_vhost.local should have similar done.
Don't forget to configure logrotate to rotate the logs.

Now you can tail the access log for all sites to figure which site or IP is affecting your server.
Very easy to spot for example wordpress or joomla bruteforces and deal with them (be that botnet or a single machine). This log can also be used as signle watch point for stuff like fail2ban (to deal with bruteforces above) or similar.

I hope this will be helpfull for someone in the future.
I wander sometimes why this is not done by default. Logs are very important for security purposes.
I am not the first one to ask for it:
http://forums.cpanel.net/f5/master-apache-access-log-352032.html
http://forums.cpanel.net/f5/global-apache-access-log-164866.html
http://forums.cpanel.net/f5/apache-access-log-file-all-accounts-150721.html
http://forums.cpanel.net/f5/there-single-apache-log-all-access-100661.html

All these were unaswered.
Another problem I found is that the WHM will delete all the access logs without archival, once the stats were processed by default. We have disabled that anti-feature.
 
  • Like
Reactions: luis_h

cPanelMichael

Administrator
Staff member
Apr 11, 2011
47,908
2,218
463
Hello :)

I am happy to see you were able to resolve the issue. Thank you for updating this thread with the solution. Note that you are welcome to submit a feature request to have this behavior utilized by default:

Submit A Feature Request

Thank you.
 

cwdd

Registered
Aug 5, 2013
1
0
1
cPanel Access Level
Root Administrator
isn't /usr/local/apache/logs/access_log already a combined log of requests? I was happy to find these:

Code:
	/usr/local/apache/logs/fpcgisock
	/usr/local/apache/logs/access_log
	/usr/local/apache/logs/error_log
	/usr/local/apache/logs/httpd.pid
	/usr/local/apache/logs/suexec_log
	/usr/local/apache/logs/ssl_scache.dir
	/usr/local/apache/logs/ssl_scache.pag
	/usr/local/apache/logs/suexec_log.offset
	/usr/local/apache/logs/suphp_log
It may also be worth looking at piped logging and ALSO this site discussing optimization

Be sure to check out the cPanel config option for piped logging, too

ed6ec3650e.png