Unable to find (or modify) log format in Apache configuration.

[Sinus Pi]

I'm trying to find - and modify - the "combined" log format Apache uses. And any other log-related settings, really.

My /etc/apache2/conf/httpd.conf file has the following log format directive:

<IfModule mod_log_config.c>
...
    LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
</IfModule>

This section seems to be properly produced by the ea4_main.default template, nothing strange about it. But, my /usr/local/apache/domlogs/domain.com log file is formatted differently:

209.299.199.299 - - [25/Nov/2018:03:10:04 -0500] "GET /url/somefile" 200 - "-" "UserAgent"

It clearly isn't using the format which starts with "%v:%p", a hostname and a port.

Then, my cpanel/userdata/username/domain.com file has the following lines:

customlog:
  -
    format: "\"%{%s}t %I .\\n%{%s}t %O .\""
    target: /usr/local/apache/domlogs/domain.com-bytes_log
  -
    format: combined
    target: /usr/local/apache/domlogs/domain.com

This seems to indicate the "combined" format for the domain.com log, but as shown above, "combined" must've been overridden... somewhere. Also, I find no trace of the domlogs/domain.com file compiled into httpd.conf anywhere that I could find. Not in httpd.conf, not in any virtualhost includes (I have some custom includes, but they're not created with a template).

So - where does Apache get the log formats, and how do I modify them? Or, more widely: how EXACTLY are the Apache configuration files built? So far I was sure it's cpanel/templates + cpanel/userdata = httpd.conf, but apparently there's more to it.

My curiosity is also piqued by some httpd.conf.work.QiHpbC20 files in /etc/apache2/conf, there ARE domain.com-related entries in them, but they aren't actively used, now are they?

 

[cPanelMichael]

Hello @Sinus Pi,

I'm trying to find - and modify - the "combined" log format Apache uses. And any other log-related settings, really.

You can modify the following options in WHM >> Service Configuration >> Apache Configuration >> Global Configuration to change the default LogFormat configuration:

LogFormat (combined)
LogFormat (common)

The discussions on the following threads offer some more insight into how to make changes to the default Apache LogFormat configuration via the command line:

SOLVED - Global Log
Apache override templates installed warning
mod_cloudflare vs mod_remoteip
Apache log format issue with EasyApache 4 (apache 2.4), Varnish and mod_remoteip

So - where does Apache get the log formats, and how do I modify them? Or, more widely: how EXACTLY are the Apache configuration files built? So far I was sure it's cpanel/templates + cpanel/userdata = httpd.conf, but apparently there's more to it.

The template files in /var/cpanel/templates/apache2_4/ determine how the LogFormat entries are populated when /scripts/rebuildhttpdconf builds the /etc/apache2/conf/httpd.conf file. Here's the default LogFormat configuration in /var/cpanel/templates/apache2_4/main.default in cPanel & WHM version 76:

[% IF enable_piped_logs -%]
<IfModule mod_log_config.c>
    LogFormat "%v:%p [% IF main.exists('logformat_combined') %][% main.logformat_combined.item.logformat_combined %][% ELSE %]%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"[% END %]" combinedvhost
    <IfModule logio_module>
        LogFormat "%v %{%s}t %I .\n%v %{%s}t %O ." bytesvhost
    </IfModule>
    LogFormat "%v:%p [% IF main.exists('logformat_combined') %][% main.logformat_combined.item.logformat_combined %][% ELSE %]%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"[% END %]" combined
    LogFormat "%v:%p [% IF main.exists('logformat_common') %][% main.logformat_common.item.logformat_common %][% ELSE %]%h %l %u %t \"%r\" %>s %b[% END %]" common
    LogFormat "%{Referer}i -> %U" referer
    LogFormat "%{User-agent}i" agent

    [%- SET splitlogsextra = '' -%]
    [%- TRY -%]
        [%- SET splitlogsconf = load_conf('/var/cpanel/conf/splitlogs.conf') -%]
        [%- IF splitlogsconf.maxopen.length %][% splitlogsextra = splitlogsextra _ " --maxopen=" _ splitlogsconf.maxopen %][% END -%]
        [%- IF splitlogsconf.buffer.length %][% splitlogsextra = splitlogsextra _ " --buffer=" _ splitlogsconf.buffer %][% END -%]
        [%- IF splitlogsconf.sslport.length %]
            [% splitlogsextra = splitlogsextra _ " --sslport=" _ splitlogsconf.sslport %]
        [%- ELSIF configured.main_port_ssl.length && configured.main_port_ssl != '443' %]
            [% splitlogsextra = splitlogsextra _ " --sslport=" _ configured.main_port_ssl %]
        [% END -%]
    [%- CATCH -%]
        [%# noop but catch is required or it dies %]
    [%- END -%]
    <IfModule logio_module>
        CustomLog "|/usr/local/cpanel/bin/splitlogs --dir=[% paths.dir_domlogs %] --main=[% wildcard_safe(servername) %] --suffix=-bytes_log[% splitlogsextra %]" bytesvhost
    </IfModule>
    CustomLog "|/usr/local/cpanel/bin/splitlogs --dir=[% paths.dir_domlogs %] --main=[% wildcard_safe(servername) %] --mainout=[% paths.file_access_log %][% splitlogsextra %]" combinedvhost
</IfModule>
[% ELSE %]
<IfModule log_config_module>
[% IF main.exists('logformat_combined') || main.exists('logformat_common') %]
    LogFormat "[% IF main.exists('logformat_combined') %][% main.logformat_combined.item.logformat_combined %][% ELSE %]%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"[% END %]" combined
    LogFormat "[% IF main.exists('logformat_common') %][% main.logformat_common.item.logformat_common %][% ELSE %]%h %l %u %t \"%r\" %>s %b[% END %]" common
    LogFormat "%{Referer}i -> %U" referer
    LogFormat "%{User-agent}i" agent
[%  ELSIF main.ifmodulemodlogconfigc.logformat.items.length -%]
[%      FOREACH dir IN main.ifmodulemodlogconfigc.logformat.items -%]
    LogFormat [% dir.logformat %]
[%      END -%]
[%  ELSE -%]
    LogFormat "%{Referer}i -> %U" referer
    LogFormat "%{User-agent}i" agent
    # NOTE: "combined" and "common" are required by WHM
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
[%  END -%]

    # access_log format can be set in WHM under 'Basic WebHost Manager® Setup'
[%  IF main.ifmodulemodlogconfigc.customlog.items.length -%]
[%      FOREACH dir IN main.ifmodulemodlogconfigc.customlog.items -%]
    CustomLog [% dir.target %] [% dir.format %]
[%      END -%]
[%  ELSE -%]
    CustomLog logs/access_log [% logstyle %]
[%  END -%]
</IfModule>
[% END %]

The /var/cpanel/cpanel.config file is used to determine if pipped logging (WHM >> Apache Configuration >> Piped Log Configuration) is enabled or disabled:

grep piped /var/cpanel/cpanel.config
enable_piped_logs=1

This section seems to be properly produced by the ea4_main.default template, nothing strange about it. But, my /usr/local/apache/domlogs/domain.com log file is formatted differently:

209.299.199.299 - - [25/Nov/2018:03:10:04 -0500] "GET /url/somefile" 200 - "-" "UserAgent"

It clearly isn't using the format which starts with "%v:%p", a hostname and a port.

That's the correct format and matches what I see on a test system with pipped logging enabled. EX:

1.2.3.4 - - [29/Nov/2018:11:00:43 -0600] "GET /favicon.ico HTTP/1.1" 404 - "-" "$USERAGENT-HERE"

If you're using mod_remoteip, then you should be able to simply browse to WHM >> Service Configuration >> Apache Configuration >> Global Configuration and change "%h" to "%a" on the following options for the desired effect of logging proxied traffic:

LogFormat (combined)
LogFormat (common)

Let me know if this helps.

Thank you.

 

[Sinus Pi]

Thanks for taking the time to respond! Indeed modifying the LogFormat "combined" setting in WHM worked as intended.

However, in case someone else has the same conundrum, let me state what exactly I learned.

These settings in Apache's httpd.conf file,

<IfModule mod_log_config.c> (yes, mod_log_config is loaded, I checked with httpd -M)
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedvhost
...
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b" common
...
</IfModule>

... ARE where the log formats are set, although with "%v:%p " prepended for the splitlogs utility to use. I made the mistake of looking for where my setting would be present literally, "%h %l %u..." - the "%v:%p" misled me into thinking this was NOT the correct place to set it. Instead, should one ever end up modifying the LogFormat's generation via ea4_main.* templates, one must remember to make sure "%v:%p " is present, otherwise splitlogs won't work.

Cheers!

 

[cPanelMichael]

Hello @Sinus Pi,

Thanks for sharing the outcome!

Instead, should one ever end up modifying the LogFormat's generation via ea4_main.* templates, one must remember to make sure "%v:%p " is present, otherwise splitlogs won't work.

We recommend using WHM >> Service Configuration >> Apache Configuration >> Global Configuration to change the default LogFormat configuration as opposed to creating a local EA4 template for that purpose.

I've created an internal request to create a tutorial on how to enable mod_remoteip on cPanel & WHM and made a note to include the caveat noted in your quote.

Thank you.

 

[Sinus Pi]

And rightfully you do recommend using WHM's configuration, but should one try to create a specific log format for just one virtualhost, by adding a new variable in cpanel/userdata/<vhostname> and modifying their EA4 templates to use it, they'll need to understand how the %v:%p part is used.