[CPANEL-31754] SPF false positive, fails on valid senders after upgrade

[Pedro Marques]

After upgrading to CentOS7 with cPanel 86.0.04 some incoming email bounce due to SPF check.

Most of those emails have a valid sender IP address that matches domain SPF record but for some reason Exim's SPF fail to match but if I run a test on numerous validators (online or command line) that same sender / spf record are correct.

From what I've noticed here's two cases where Exim's SPF fails:

- SPF check does not follow the 'a' tag to the respective IP address but if the record contains ip4:xxx (with the same IP as the A record) it passes;
- The 'exists:' tag is not processed correctly PTR records with wildcard records

Here's some proof / test so someone can take a look at it.

Example rejection:

H=smtp1.example.pt (anubis04.example.pt) [<IPAddress>]:11970 X=TLS1.2:ECDHE-RSA-AES256-GCM-SHA384:256 CV=no F=<******@example.pt> rejected RCPT <******@******>: SPF: <IPAddress> is not allowed to send mail from example.pt

SPF record for example.pt:

v=spf1 a:smtp1.example.pt a:smtp2.example.pt a:smtp3.example.pt a:smtp4.example.pt a:smtp5.example.pt a:newsletter.example.pt include:spf.protection.outlook.com -all

Sender IP address matches the SPF record:

smtp1.example.pt. 21600 IN A <IPAddress>

How can I tweak / fix this? Is this check some external call?

Thanks,
PM

 

[cPanelLauren]

Can you show the entire exim transaction? You should be able to see the MessageID in the logs and then run something like the following with it:

exigrep <MessageID> /var/log/exim_mainlog

Another thing I noticed is that while the domain has a valid A record the domain doesn't actually have a valid txt record in place:

dig txt smtp1.example.pt +short

Has no return

 

[Pedro Marques]

Can you show the entire exim transaction? You should be able to see the MessageID in the logs and then run something like the following with it:

Hi, thanks for reply, domain names were edited by forum review.

 

[Juan R. Pozo]

We are seeing the same here, email is being rejected due to false SPF positives. It's only happening on the two servers that have been upgraded to 86.0.4. The other servers that are still on 84.0.21 are working correctly.

For example:

2020-02-25 10:39:46 SMTP connection from [<LocalIPAddress>]:30200 (TCP/IP connection count = 39)
2020-02-25 10:40:27 H=smtp1.domain.cat [<LocalIPAddress]:30200 F=<******@domain.cat> rejected RCPT <******@******.com>: SPF: <LocalIPAddress> is not allowed to send mail from domain.cat
2020-02-25 10:40:32 SMTP connection from smtp1.domain.cat [<LocalIPAddress>]:30200 closed by QUIT

The IP address corresponds to on of the MX records which should be allowed by the SPF record:

domain.cat. 1562 IN TXT "v=spf1 mx ?a:postari.txtdomain.cat ?ip4:<IPAddress> -all"

 

[cPanelLauren]

Hi, thanks for reply, domain names were edited by forum review.

I removed the domain names as we remove all personally identifying information from posts, this is discussed here: In Progress - Guide To Opening An Effective Forums Thread. Regardless though I can see your SPF record now:

"v=spf1 a:smtp1.example.pt a:smtp2.example.pt a:smtp3.cp.example a:smtp4.example.pt a:smtp5.cp.pt a:newsletter.example.pt include:spf.protection.outlook.com -all"

You're matching all domains and IP's but the originating domain, example.pt, does this persist if you add a:example.pt to the record?


I noticed both of you are hard failing addresses that don't match with "-all" while testing issues with the SPF you may want to soft fail instead "~all"

• - Fail, an IP that matches a mechanism with this qualifier will fail SPF.
• ~ SoftFail, an IP that matches a mechanism with this qualifier will soft fail SPF, which means that the host should accept the mail, but mark it as an SPF failure.

The qualifier in this instance being "all"


@Juan R. Pozo your SPF record specifically does not include anything useful for your mail sending domain besides the mx, but it also does not include the domain or the domain's IP address.

The qualifier "?" is neutral, an IP that matches the following mechanism with this qualifier will neither pass or fail SPF - followed by a hard fail for anything that doesn't match.

 

[Pedro Marques]

I removed the domain names as we remove all personally identifying information from posts, this is discussed here: In Progress - Guide To Opening An Effective Forums Thread. Regardless though I can see your SPF record now:

Ok, I understand now

You're matching all domains and IP's but the originating domain, example.pt, does this persist if you add a:example.pt to the record?

Like I said, the host is implicit / match with the original SPF record through the A Record.

Example: the host sending email is smtp1.example.pt [XXX.XXX.XXX.XXX] and the SPF contains "a:smtp1.example.pt" but cPanel say it's not valid to send email but, if I add to the SPF record "ip4:XXX.XXX.XXX.XXX" cPanel will check correctly the IP address and accept the message.

Another thing I've noticed is the "a" tag alone in the SPF ('v=spf1 a -all') , which should check the A record for the host used in the email, cPanel won't check it and fail every time.

It looks like SPF check is not resolving addresses correctly converting names in SPF records to real IP address used by the sender.

Thanks.

 

[Juan R. Pozo]

Hi Lauren, I don't understand your reply, sorry. The IP address of the sender host matches the mx mechanism of the domain's SPF record:

"This mechanism matches if <ip> is one of the MX hosts for a domain name."

as is the case here. Therefore it is a match and the check should not fail.

There are many other false positives, this was just the first report I got yesterday.

 

[cPanelLauren]

What I'm trying to reconcile here is that there have been no significant changes to SPF validations which would cause this as far as I am aware. The update to v86 had no changes to exim or SPF per the release notes and changelogs here: Change Logs - Change Logs - cPanel Documentation - further to that I don't have any open cases that indicate a change to SPF. Identifying where the issue is coming from would be key but I'm not seeing anything thus far that would tell me that this isn't a misconfiguration.


@Pedro Marques Looking at the domain's DNS records I found that it resolves to two separate IP addresses - one of which does not return any DNS records for the domain. Which is why I was unable to see the SPF yesterday.

[root@server templates]# dig a +short example.pt
80.251.XXX.XXX
81.90.XX.XXX
[root@server templates]# dig txt +short @80.251.XXX.XXX example.pt
"v=spf1 a:smtp1.example.pt a:smtp2.example.pt a:smtp3.example.pt a:smtp4.example.pt a:smtp5.example.pt a:newsletter.example.pt include:spf.protection.outlook.com -all"
"OJsOzrrhyuyFNpPNraj9vrlhH9EcI2rVOJjsFERpednPBReUj4JcoTLjSMD0wzbEDb5IJwekTBV6zXVHReWubA=="
[root@server templates]# dig txt +short @81.90.XX.XXX example.pt
;; connection timed out; no servers could be reached

As far as the A record goes, I can't rule out an issue with the configuration with one of those IP's timing out returning records for the domain if you could humor me for one second and ensure that DNS resolution for the domain is working properly on all IP's as well as add a or +a I'd like to see if that moves you in the right direction.


@Juan R. Pozo

mx in that record is the only thing being looked at because of the "?" qualifier you're using for a and ip4 - if you change mx to +mx does this resolve anything? (I realize that it should be fine with mx as mx should be equivalent to +mx but I'd still like to know.

 

[Juan R. Pozo]

@cPanelLauren

If I use "v=spf1 mx -all" it passes, if I use "v=spf1 mx ?ip4:a.b.c.d -all" (or "v=spf1 +mx ?ip4:a.b.c.d -all") it fails.

It seems to be an Exim bug that was fixed recently, please see here for details: https://bugs.exim.org/show_bug.cgi?id=2499

 

[cPanelLauren]

I'm opening a case for the exim bug report you found and thank you for that, it seems to be related to having multiple (> 1) MX references and DNS RR's not being returned for one of them

When a TXT SPF record contains two MX references, and the second reference returns no DNS RRs, Exim erases the results retrieved from the first lookup, treating it as also null. This causes the SPF ACL to return "fail" even though the first MX clause should short-circuit the result as "pass".

it seems that based on that the ip4: reference should suffice, which leads me to wonder, I'm curious, why you are using the "?" qualifier in that record and not the "+"? I'm unaware of a reason for that in this context and genuinely curious. You could not include the ip4: and -all since it's a neutral result could you not? Keep in mind I fully understand that the initital mx match should produce a success not a fail. I'm primarily just curious and wondering if it could be a potential workaround until this is resolved.

 

[Juan R. Pozo]

I really don't know, we don't administer that domain. My guess is that they have disabled those mechanisms temporarily.

I could replicate the error in one domain I own just by adding another mechanism after the mx mechanism. The bug is not really about its initial description only. Here https://bugs.exim.org/show_bug.cgi?id=2499#c26 you can see that a patch is submitted which fixes several other cases, one of which I think is similar to the dot cat domain in my report.

 

[cPanelLauren]

I've opened CPANEL-31754 for this issue thanks to your information. I chose to go with an internal case rather than an upstream case due to the fact that exim does have a patch for this but it's not available in CentOS upstream

I fully understand how this is replicable now. I did clarify I understand that the initial match on mx should be a success - I was just thinking as far as my own curiosity and potentially a workaround until it's resolved.

 

[Juan R. Pozo]

Thanks @cPanelLauren, in the meanwhile we'll be invoking spfquery from exim.conf instead of using the built-in spf facility.

 

[Pedro Marques]

@Pedro Marques Looking at the domain's DNS records I found that it resolves to two separate IP addresses - one of which does not return any DNS records for the domain. Which is why I was unable to see the SPF yesterday.

[root@server templates]# dig a +short example.pt
80.251.XXX.XXX
81.90.XX.XXX
[root@server templates]# dig txt +short @80.251.XXX.XXX example.pt
"v=spf1 a:smtp1.example.pt a:smtp2.example.pt a:smtp3.example.pt a:smtp4.example.pt a:smtp5.example.pt a:newsletter.example.pt include:spf.protection.outlook.com -all"
"OJsOzrrhyuyFNpPNraj9vrlhH9EcI2rVOJjsFERpednPBReUj4JcoTLjSMD0wzbEDb5IJwekTBV6zXVHReWubA=="
[root@server templates]# dig txt +short @81.90.XX.XXX example.pt
;; connection timed out; no servers could be reached

As far as the A record goes, I can't rule out an issue with the configuration with one of those IP's timing out returning records for the domain if you could humor me for one second and ensure that DNS resolution for the domain is working properly on all IP's as well as add a or +a I'd like to see if that moves you in the right direction.

Sender from domain @example.pt. SPF record obtained from example.pt which contains the tag 'a:smtp1.example.pt' as allowed host to send from this domain.
When the sender reach Exim, the IP address of smtp1.example.com is rejected because SPF did not match the IP address of ANY 'a' tags in SPF record for that domain. If the SPF record states the same host IP as 'ip4:XXX.XXX.XXX.XXX' it passes.

Proof is in the original post that was shadowed but I can easily replicate with other examples if you want and send you with real data.
Had to disable SPF check because it was denying a lot of good, valid messages.

 

[cPanelLauren]

Sender from domain @example.pt. SPF record obtained from example.pt which contains the tag 'a:smtp1.example.pt' as allowed host to send from this domain.
When the sender reach Exim, the IP address of smtp1.example.com is rejected because SPF did not match the IP address of ANY 'a' tags in SPF record for that domain. If the SPF record states the same host IP as 'ip4:XXX.XXX.XXX.XXX' it passes.

Proof is in the original post that was shadowed but I can easily replicate with other examples if you want and send you with real data.
Had to disable SPF check because it was denying a lot of good, valid messages.

Thanks! I agree, the second notice of this was a bit easier to prove in this instance, any time I'm looking at a potential defect I need the most reliable and straightforward way to replicate it, in the instance of yours there were a few variables that made it less easy to be replicated.

None the less, as you can see I was able to get a case opened on the information provided, furthermore it's a known issue with exim as is noted in previous responses. The case I opened was to apply the patch that exim has put out for this. I'll update here when there is more information.

Thanks @cPanelLauren, in the meanwhile we'll be invoking spfquery from exim.conf instead of using the built-in spf facility.

If you are able to provide the workaround you employ here, I'm sure it would be much appreciated.

 

[cPanelLauren]

Hello @Juan R. Pozo and @Pedro Marques

Can you both let me know if you have any custom ACL's in the exim configuration? You'd find these at /etc/exim.conf.local.

I ask this specifically because though the functionality to do this exists in exim, and we will still be adding the patch based on the case I opened, cPanel's version of Exim does not by default do any SPF denials, instead that's handled by SpamAssassin.

 

[Hedloff]

We have the same issue.
We have some custom ACL's because of spamexperts. But for SPF check we have this in "Exim Config Manager" -> Advanced Editor -> custom_begin_mailauth:

deny message = SPF: $sender_host_address is not allowed to send mail from $sender_address_domain
    !condition = ${if match_ip{$sender_host_address}{iplsearch;/etc/trustedmailhosts}{1}{0}}
    spf = fail

Any idea when you will push a fix for this?
What is the easiest workaround for this?

 

[Pedro Marques]

Hello @Juan R. Pozo and @Pedro Marques
Can you both let me know if you have any custom ACL's in the exim configuration? You'd find these at /etc/exim.conf.local.

Hi, the only custom config I had was the SPF in custom_begin_mailauth, but I had to remove it as it was blocking too many messages from hosts with valid SPF records.

# BEGIN INSERT spf_bl
deny message = SPF: $sender_host_address is not allowed to send mail from $sender_address_domain
    spf = fail
# END INSERT spf_bl

Thanks.

 

[cPanelLauren]

That's what I'd figured but had wanted to confirm. For all instances of this, we've seen so far, it's due to a custom ACL being present that validates SPF records + exim with the version that fails to properly perform SPF lookups as a result of the Exim case. I have the patch included in the internal case I opened, exim should be able to facilitate these DNS lookups successfully whether or not we provide a mechanism for it.

I believe a similar issue is why we originally removed the original setting to reject SPF failures from the exim configuration (a long time ago). We rely on SpamAssassin for that functionality now and it does provide quite a bit of configurability though won't perform a rejection at SMTP time for SPF.

 

[Juan R. Pozo]

Hi, this is our workaround, in case it may help someone:

domainlist skip_spf_check_domains = lsearch;/etc/skipspfcheckdomains

[...]

# BEGIN INSERT custom_begin_mailauth

warn set acl_m_spfquery = ${run{/usr/lib/exim/bin/spfquery -sender $sender_address -ip $sender_host_address -helo $sender_helo_name}{$value}}
     set acl_m_spfheader = ${if match{$value}{Received-SPF: (.+)}{$1}}

# If SPF result is fail then $runrc is 3

deny condition = ${if == {$runrc}{3}}
     !sender_domains = +skip_spf_check_domains
     message = SPF: $acl_m_spfheader

warn condition = ${if == {$runrc}{3}}
     sender_domains = +skip_spf_check_domains
     log_message = Ignoring SPF result: $acl_m_spfheader

warn !condition = ${if == {$runrc}{3}}
     condition = ${if def:acl_m_spfheader}
     add_header = :at_start:Received-SPF: $acl_m_spfheader

# END INSERT custom_begin_mailauth