mod_remoteip and correct ip in vhost access logs

[internetfab]

Good morning from Sweden

We've decided to use mod_remoteip instead of mod_rpaf to translate the XFF header (apache 2.4)
However, mod_remoteip uses the %a log format instead of the %h to get the client IP. If there is no XFF header then %a (or if it's not trusted) then %a equals %h.

So, this is fixable in httpd.conf by changing these %h into %a.

<IfModule mod_log_config.c>
    LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedvhost
    LogFormat "%v %{%s}t %I .\n%v %{%s}t %O ." bytesvhost
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    LogFormat "%{Referer}i -> %U" referer
    LogFormat "%{User-agent}i" agent

    CustomLog "|/usr/local/cpanel/bin/splitlogs --main=box01.oderland.com --suffix=-bytes_log" bytesvhost
    CustomLog "|/usr/local/cpanel/bin/splitlogs --main=box01.oderland.com --mainout=/usr/local/apache/logs/access_log" combinedvhost

</IfModule>

so they'll be like this:

<IfModule mod_log_config.c>
    LogFormat "%v:%p %a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedvhost
    LogFormat "%v %{%s}t %I .\n%v %{%s}t %O ." bytesvhost
    LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%a %l %u %t \"%r\" %>s %b" common
    LogFormat "%{Referer}i -> %U" referer
    LogFormat "%{User-agent}i" agent

    CustomLog "|/usr/local/cpanel/bin/splitlogs --main=box01.oderland.com --suffix=-bytes_log" bytesvhost
    CustomLog "|/usr/local/cpanel/bin/splitlogs --main=box01.oderland.com --mainout=/usr/local/apache/logs/access_log" combinedvhost

</IfModule>

However, after distilling and rebuilding httpdconf, one of the entries above reverts to %h:

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedvhost

That's of course the logformat that controls virtual host logs.

I've had this issue up with cPanel twice already during the last year. They couldn't solve it and asked me to file a feature request.

I've since added the whole part of the config in /usr/local/apache/conf/includes/pre_virtualhost_global.conf to overwrite the incorrect one, but it feels like a hack (since cPanel could change the template in the future or decide to move the splitlogs binary).

Has anyone else solved it in a better way? Care to share how?

 

[DavidN.]

Hi there,

Can you check the directive in /var/cpanel/conf/apache/main ? By default, it looks like this:

    ifmodulelogiomodule:
      logformat:
        directive: logformat
        items:
          -
            logformat: "\"%h %l %u %t \\\"%r\\\" %>s %b \\\"%{Referer}i\\\" \\\"%{User-Agent}i\\\" %I %O\" combinedio"
    logformat:
      directive: logformat
      items:
        -
          logformat: "\"%h %l %u %t \\\"%r\\\" %>s %b \\\"%{Referer}i\\\" \\\"%{User-Agent}i\\\"\" combined"
        -
          logformat: "\"%h %l %u %t \\\"%r\\\" %>s %b\" common"

You should be able to just change %h to %a, then run /scripts/rebuildhttpdconf and have it do what you want. Please let me know if that doesn't work. Please be careful editing this file; it's YAML, which is very sensitive to whitespace changes.

Edit: I fail at formatting. Make sure you preserve the indentation in this file, or you may get a broken Apache configuration.[/CODE]

 

[internetfab]

Hi David!
Thanks for getting back to me. I've been down this road before
The

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedvhost

is missing from the yaml conf.

I just realized I sent you guys a feature request January 2015.

Here's what I wrote in the ticket about it:

I found the source - it would seem that the LogFormat line is hardcoded in: /usr/local/cpanel/Cpanel/AdvConfig/apache.pm

Therefor it leaves no way for us to change the LogFormat to be used with mod_remoteip module for domlogs (using split logs).

I've made an diff which i'm sending to you below. This is in no way a complete patch as I'm just changing one hard coded value for another, but I just want to show what we need to change for mod_remoteip to work for domlogs.

The problem otherwise is that the IP of the proxy will get blocked in the firewall due to CSF/LFD or cpHulk when we're checking modsec or our own LFD modules.

Please - make that logformat line configurable in main.default or local in /var/cpanel/conf/apache/ in case it exists.. otherwise use default if you want to.

Steps to reproduce:

# [/usr/local/cpanel/Cpanel/AdvConfig]# diff -c apache.pm apache.pm.new
*** apache.pm 2015-01-28 14:07:43.388000008 +0100
--- apache.pm.new 2015-01-28 14:05:20.870000009 +0100
***************
*** 371,377 ****
# Add new log formats.
$logconf->{'logformat'}->{'items'} = [
{
! 'logformat' => q["%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedvhost],
},
(
$has_logio
--- 371,377 ----
# Add new log formats.
$logconf->{'logformat'}->{'items'} = [
{
! 'logformat' => q["%v:%p %a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedvhost],
},
(
$has_logio

 

[internetfab]

Ticket id: 6031689

 

[internetfab]

Any updates on this @cPDavidN? Is it a "feature" or a bug?

 

[DavidN.]

Hi Internetfab,

I apologize for missing your reply. The way that Apache templates and those .yaml files interact is complex and strange and, since they're generated on the server itself, comparisons between machines aren't helpful. (I've been dealing with different aspects of this subsystem for the last couple of days, so I've gotten some new insights into why my last reply was not so helpful.)

Would it be possible for you to re-open your ticket and provide access to an affected server? I'd like to examine your configuration and see if I can come up with something that will work for you long-term, which will survive cPanel updates, distiller runs, and Apache configuration rebuilds.

 

[santrix]

We also see this problem. on 11.48.4 (build4). From /var/cpanel/conf/apache/main

         logformat: "\"%v:%p %a %l %u %t \\\"%r\\\" %>s %b \\\"%{Referer}i\\\" \\\"%{User-Agent}i\\\"\" combinedvhost"
        -
          logformat: "\"%v %{%s}t %I .\\n%v %{%s}t %O .\" bytesvhost"
        -
          logformat: "\"%a %l %u %t \\\"%r\\\" %>s %b \\\"%{Referer}i\\\" \\\"%{User-Agent}i\\\"\" combined"
        -
          logformat: "\"%a %l %u %t \\\"%r\\\" %>s %b\" common"
        -
          logformat: "\"%{Referer}i -> %U\" referer"
        -
          logformat: "\"%{User-agent}i\" agent"

After /scripts/rebuildhttpdconf we see in the httpd.conf

<IfModule mod_log_config.c>
    LogFormat "%v:%p %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combinedvhost
    LogFormat "%v %{%s}t %I .\n%v %{%s}t %O ." bytesvhost
    LogFormat "%a %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%a %l %u %t \"%r\" %>s %b" common
    LogFormat "%{Referer}i -> %U" referer
    LogFormat "%{User-agent}i" agent

    CustomLog "|/usr/local/cpanel/bin/splitlogs --main=orcus.uksrv.co.uk --suffix=-bytes_log" bytesvhost
    CustomLog "|/usr/local/cpanel/bin/splitlogs --main=orcus.uksrv.co.uk --mainout=/usr/local/apache/logs/access_log" combinedvhost

</IfModule>

The combinedvhost entry, for some reason totally ignores the line entry in /var/cpanel/conf/apache/main - I've inserted extra spaces etc., and none of it has any effect on the LogFormat directive for combinedvhost

 

[cPanelMichael]

The combinedvhost entry, for some reason totally ignores the line entry in /var/cpanel/conf/apache/main - I've inserted extra spaces etc., and none of it has any effect on the LogFormat directive for combinedvhost

Try using /var/cpanel/conf/apache/local instead of /var/cpanel/conf/apache/main in order to implement custom changes. You will need to create "/var/cpanel/conf/apache/local" if it does not yet exist.

Thank you.

 

[santrix]

Try using /var/cpanel/conf/apache/local instead of /var/cpanel/conf/apache/main in order to implement custom changes.

While both files look like YAML, they appear to provide quite different functions

/var/cpanel/conf/apache/local - appears to hold the settings from WHM »Service Configuration »Apache Configuration »Global Configuration

/var/cpanel/conf/apache/main - appears to hold the settings that are directly translated into httpd.conf

I'm taking a wild guess, but I doubt adding the following to our pre-existing local file is going to help - or would probably break a rebuild (I haven't tested)

---
defaultvhost:
  userdirprotect: ''
main:
  ifmodulemodlogconfigc:
    customlog:
      directive: customlog
      items:
        -
          format: --main=ourhost.co.uk --suffix=-bytes_log" bytesvhost
          target: "\"|/usr/local/cpanel/bin/splitlogs"
        -
          format: --main=ourhost.co.uk --mainout=/usr/local/apache/logs/access_log" combinedvhost
          target: "\"|/usr/local/cpanel/bin/splitlogs"
    logformat:
      directive: logformat
      items:
        -
          logformat: "\"%v:%p %a %l %u %t \\\"%r\\\" %>s %b \\\"%{Referer}i\\\" \\\"%{User-Agent}i\\\"\" combinedvhost"
        -
          logformat: "\"%v %{%s}t %I .\\n%v %{%s}t %O .\" bytesvhost"
        -
          logformat: "\"%a %l %u %t \\\"%r\\\" %>s %b \\\"%{Referer}i\\\" \\\"%{User-Agent}i\\\"\" combined"
        -
          logformat: "\"%a %l %u %t \\\"%r\\\" %>s %b\" common"
        -
          logformat: "\"%{Referer}i -> %U\" referer"
        -
          logformat: "\"%{User-agent}i\" agent"
sections:
  ifmodulemodlogconfigc: <ifmodule  mod_log_config.c>
supported:
  mod_log_config: 1

 

[terraGirl]

Is there any update on this? We're using nginx as reverse proxy with Apache 2.4 / mod_remoteIP and can't get it to work, it keeps logging the server IP which makes access controls like eg WordPress brute force attack prevention etc useless. Any help in getting mod_remoteIP to work correctly with cPanel would be much appreciated.

 

[santrix]

For the time being - the only way around this is to directly edit the following files...

sed -i.original -e 's/\(logformat.*\)%h\(.*\)/\1%a\2/' /var/cpanel/conf/apache/main
sed -i.original -e 's/\(.*\)%h \(.* combinedvhost.*\)/\1%a \2/' /usr/local/cpanel/Cpanel/AdvConfig/apache.pm

This will ensure that %h is replaced with %a for the combinedvhost LogFormat entry (in addition to the other LogFormat entries) in httpd.conf

This will get overwritten after a upcp, so you may want to put these instructions in a script and add that script to /usr/local/cpanel/scripts/postupcp

 

[terraGirl]

This will get overwritten after a upcp, so you may want to put these instructions in a script and add that script to /usr/local/cpanel/scripts/postupcp

Thank you! Do I just put the 2 lines as provided into postupcp or do I need to write my own script? Sorry for the newbie query, first time I'm messing with cPanel settings. I've tried to just put the 2 lines into postupcp but that did not fix the IP error.

 

[santrix]

It's probably more tidy to put them into a short bash script and pop that into /usr/local/cpanel/scripts/postupcp instead.

If you want the changes to take effect immediately then you will need to run those two sed commands, and then run /scripts/rebuildhttpdconf, and /scripts/rebuildnginxvhost, and then service httpd restart.

It should be noted that this technique has been tested and works with 11.54 (Build 19), but you know how cPanel change things ;-)

 

[terraGirl]

Hi Santrix, Thank you! I'm on WHM 54.0 (build 19), and the fix doesn't work for me.

 

[santrix]

Then, check /usr/local/apache/conf/httpd.conf for each of the LogFormat directives to ensure that the %h options have been converted to %a.

If so, then it may be that your nginx reverse proxy isn't adding the X-Real-IP header for the correct server IP addresses, or maybe you aren't loading the remoteip module... Depending on your implementation this could be in various places.

Check it's loaded by running

/usr/local/apache/bin/apachectl -M

and look for remoteip_module in the output. If it is loaded then you need to ensure that your server's local IP addresses are configured as RemoteIPInternalProxy's with the following directives - these could be in your httpd.conf or in an include file depending on your setup. You should fine one or more lines like this, with your IP addresses listed:

RemoteIPInternalProxy x.x.x.x

If you have all that, then it's going to be quicker to let cPanel take a look

 

[terraGirl]

Hi Santrix,

Thank you, finally got it working! I think I was making too many changes and not properly restarting in between tests. The settings you gave me work perfectly now

And if anyone else is having issues, this setting was also important:

RemoteIPHeader X-Real-IP
RemoteIPInternalProxy 127.0.0.1
RemoteIPInternalProxy ..... [add IPs]

I tried this with "RemoteIPHeader X-Forwarded-For" first but "RemoteIPHeader X-Real-IP" is the one that works.

and in the remoteip configuration I had:

#proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Real-IP  $remote_addr;
real_ip_header     X-Forwarded-For;
real_ip_recursive on;

A quick way to test output is to show the last 50 lines of the access logs:

tail -50 /usr/local/apache/domlogs/USERNAME/DOMAIN.COM

Thank you Santrix for your help, much appreciated! Incidentally when I asked cPanel for help, they simply changed %a back to %h Glad about this forum.

Best regards, Edith

 

[AnthonyFrancis]

The custom_opt_mod-remoteip.tgz seems to be missing from the downloads. The download link 404's.

 

[AnthonyFrancis]

Thanks for fixing the link (whomever did it)!

 

[cPanelMichael]

Thanks for fixing the link (whomever did it)!

I'm happy to see it's working well. It may have been a temporary server issue as I don't see any reports that a case was opened to address the problem. Let us know if you encounter any additional issues with the download.

Thanks!