PHP mail() function hosted on causes email to end up in SPAM. Solution?

[jazee]

Email send from the PHP mail() function hosted on a Cpanel account frequently ends up in SPAM (all the time for @outlook.com, some of the time for @gmail.com)

I've determined it is NOT due to the content of the message but due to something in the headers. My best guess is that it's either because there's a header field indicating the email was sent by a PHP script, or it's the fact that no matter if you set the From: Reply-To: and Return-Path: to the same email address, the SMTP server is always showing the email was sent from {cpanel account username}@servername.serverdomain.com and this does not match the From/Reply-To address and it appears it overrides any Return-Path address I use to the {cpanel account username}@servername.serverdomain.com

As you know Web apps can send email on behalf of a user where the user doesn't have an email address hosted on the same domain that the application is running from. So the server may be serverdomain.com and the user's email might be [email protected]

Examining the headers of the test script or the emails sent from, the SMTP server is always showing the email was sent from [email protected] Part of the header after coming through to Gmail shows this:

Return-Path: <[email protected]>
Received: from server1.serverdomain.com (server1.serverdomain.com. [XXX.XXX.XXX.XXX])
by mx.google.com with ESMTPS id w7si15164878pgi.491.2020.09.22.16.51.14
for <[email protected]>
(version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128);
Tue, 22 Sep 2020 16:51:14 -0700 (PDT)
Received-SPF: pass (google.com: domain of [email protected] designates XXX.XXX.XXX.XXX as permitted sender) client-ip=XXX.XXX.XXX.XXX;
Authentication-Results: mx.google.com;
spf=pass (google.com: domain of [email protected] designates XXX.XXX.XXX.XXX as permitted sender) smtp.mailfrom=[email protected]
Received: from cpanel-account-username by server1.serverdomain.com with local (Exim 4.93) (envelope-from <[email protected]>) id 1kKs41-00014X-4u for [email protected]; Tue, 22 Sep 2020 16:51:13 -0700
To: [email protected]
Subject: Something
X-PHP-Script: suitecpanelhostdomain.com/cms/index.php for XXX.XXX.XXX.XXX
X-PHP-Originating-Script: 1006:test_script.php
From: From Name <[email protected]>
Reply-To: [email protected]
MIME-Version: 1.0
Content-Type: text/html; charset=ISO-8859-1
X-Priority: 3
X-Mailer: PHP7.3.22

If I was able to change the header so it thinks the email was sent from [email protected] instead of [email protected] I don't think that would work as the server1.serverdomain.com's IP address would not match the MX or SPF records for someotherdomain.com since the cpanel account's domain does not handle email for that application user's email addresses domain.

Some coders out there running PHP on CPanel must have run into this before and wondering if anyone found any solutions?

 

[sparek-3]

This has notoriously been a target of discussion. One that I really never bought into. People will tell you to avoid using PHP's mail() function and instead use something like PHPMailer to connect to your server's localhost through SMTP and send the message that way.

Maybe they're right, but my opinion is that people just don't understand what they are doing and it's easier to just regurgitate what they've been told.

What parameters are you passing to the mail() function in your PHP code?

Are you specifying anything for the 5th parameter to the mail() function?

You need to understand that there's a difference between the header From address (From:) and the envelope-sender address (some times called the return-path or sender from address).

When you initiate an SMTP transaction (whether that's an SMTP transaction between your script and your localhost or between your server and Gmail's mail server) after the HELO/EHLO (or after STARTTLS ... which probably every modern mail server supports now) the client will issue a MAIL FROM command. This MAIL FROM is the envelope-sender. As far as the email server MTA (Exim, Postfix, Sendmail, etc) is concerned, this envelope-sender is who sent the message.

The header From address isn't given until the SMTP transaction is in the DATA part. At that point the MTA doesn't distinguish anything. Everything is just DATA to it at this point.

SPF, DKIM, and DMARC at the MTA level is all based on the envelope-sender address.

In the example you gave

Return-Path: <[email protected]>

This is the envelope-sender. So in terms of SPF, DKIM, and DMARC the receiving mail server is only going to pay attention to the domain name - server1.serverdomain.com.

When a script uses PHPMailer for SMTP localhost sending, it's probably specifying a real email address on the specific domain name as the MAIL FROM, so the envelope-sender (or Return-Path) might become [email protected]

Now, typically you can force a direct sendmail execution of mail to change the envelope-sender by using the 5th parameter of the mail() function in PHP with something like:

mail($to, $subject, $message, "From: [email protected]\r\n", "[email protected]");

This will change the resulting headers that you receive at the Gmail account to read:

Return-Path: <[email protected]>

which will then allow SPF, DKIM, and DMARC to all be based off of someotherdomain.com - which is probably what you want.

The envelope-sender (or Return-Path) is also sometimes called a bounce address. Whatever address is set here is where bounceback messages are going to be sent. So if your script sends an email to a gmail account that doesn't exist, then Gmail is going to bounce the message back to whatever email address is listed here. That might be [email protected] or it might [email protected]

 

[jazee]

Hurray! Finally someone that appears to understand what I'm talking about and knows what they are talking about. Yes, I just didn't figure out yet how to override the envelope-sender-address so I'll give that 5th parameter try. However that will probably means it will no longer pass the SPF and DKIM tests.

I did try using PHPMailer but it made no difference. The SPAM email test service was giving me a 0.7 out of 10 for various reasons. I got it up to an 9 with the only thing remaining is getting DKIM to work. The suggested txt record for DKIM though that Cpanel shows isn't working for some reason. But even at 9/10 still going to spam for outlook.com. The envelope-sender-address is now matching the From. But as far as SPF this means I'd need to add SPF records for all the domains of all the users using the app I'm assuming.

It makes it through to Gmail no problem. But I'm wondering it Outlook has sort of blacklisted the sender since I've sent several emails now from the same server and they all go to spam and Gmail maybe in the past it was spam and I said it wasn't but my experience with gmail is that does not whitelist the sender for all future emails. I'm going to test Yahoo and a different outlook account. If it comes through on Yahoo, I may just call it a day and be happy I got the mailcheck score from 0.7/10 to 9/10.

 

[jazee]

Yep. Yahoo works fine. Outlook.com is just super finicky. Middle finger at you Microsoft! I wonder for outlook.com if it's the X-Mailer: PHPMailer 5.2.25 in the header. I've read some spam filters don't like PHP as the mailer.

 

[jazee]

Just FYI. After tons more testing I decided to try some 3rd party gateway services. I tested mailgun.org, clicksend..com and sendinblue.com. There are many others but I didn't like their pricing structure. Of the three, only sendinblue.com got the emails to the inbox instead of spam. It's free for up to 300 emails a day. Problem solved.

 

[sparek-3]

However that will probably means it will no longer pass the SPF and DKIM tests.

Well, it depends on what you set your envelope-sender to. If the domain name associated with that envelope-sender has an SPF record (that matches the sending server) and DKIM records (that again are set up on the server that is sending the message) then the message would pass these tests once it reaches the recipient server.

But even at 9/10 still going to spam for outlook.com.

All major mail service providers, but for whatever reason especially Microsoft, all have their own algorithms for calculating spam. What these algorithms are and how the function is anybody's guess. Not to get too dystopian, but the days of extra-email between different providers is going to continue to get more problematic as every email provider increases their anti-spam systems. With Hotmail to Hotmail messages, Microsoft is able to see both the sender and the receiver. With Gmail to Hotmail, Microsoft can only see the message at it's receiving end.

I wonder for outlook.com if it's the X-Mailer: PHPMailer 5.2.25 in the header.

You should be able to remove the extra headers that PHPMailer adds. Will it help? Who knows. Again just points to the secretive nature of the anti-spam measures these large email service providers use.

 

[SamuelM]

Hello @jazee

But I'm wondering it Outlook has sort of blacklisted the sender since I've sent several emails now from the same server and they all go to spam

Typically if your mail is blocked by Microsoft's mail services, you would receive a bounceback with language such as the following:

Unfortunately, messages from [$yourip] weren't sent. Please contact your Internet service provider since part of their network is on our block list

If you aren't receiving bounceback messages and the mail is simply being marked as spam, that makes it much harder to troubleshoot, because as sparek-3 explained, Microsoft have their own algorithms for calculating spam. If you haven't already reviewed the following page, it might help to provide some basic guidelines for troubleshooting the issue:

https://sendersupport.olc.protection.outlook.com/pm/troubleshooting.aspx

Although, at this point it seems as though you were able to establish a workaround with the Sendinblue smarthost.

 

[jazee]

I did some more testing and although I can't say 100% for sure, I believe the lynchpin is sending not only with a valid SPF record for the domain but also a DKIM record. DMARC made no difference. When you use 3rd party hosts like Sendinblue, they send via their domain and use their own records (there's also a chance many if these 3rd party senders are 'whitelisted' with the large email providers). But on SendInBlue, you can optionally authenticate your domain and setup SPF and DKIM records in your DNS which Sendinblue provies so the email doesn't come into Gmail showing Your Name with your reply address but 'sent via sendinblue.com' appears next to the senders name.

With a little more work I think I could probably pass the Outlook.com spam filter sending directly from my server. However, I'm not going to because I really like the nice web interface of Sendinblue.com that gives me statistics and logs of all the emails being sent, whether they bounced, if they were delivered, if they were opened, if they were clicked on and these are NOT marketing campaign emails, these are transactional emails generated to update workers and customers about status of their service order. Also when you use a 3rd party service, you can insure yourself against one of your domains on your server getting the server'ss IP blacklisted because of one of your customers sending too much spam. That way you can still send email direct from your server when desired in the future.

 

[apknerd]

The same thing is happening with me on my website: apknerd.com. I basically have a newsletter subscription but my mail() function won't send a confirmation mail to the newly registered people.
Would be great if someone can just go and test out the functionality