Global Email Filters - Body Bug

[omidsolo]

Howdy,
I'm so welcome to the forum
I'm here to ask if I'm not the only one having this issue:
Creating 2 global filters, Russian & Chinese
In each filter I enter the most used chars in each language and set if Subject or Body contains any of the, fail with a message!
For Russian I enter both lower & upper case in separate fields.
BTW, they all work just fine for Subject but NOT for Body!
You can fill the UTF-8 email body with all of blocked characters but it's not filtered and I receive those damn SPAM emails!
Seems bug to me! Advise?! cPanel 78.0.21. Site owner, no root access.

 

[omidsolo]

I found something, ain't wrong, just don’t know if this is a bug in cPanel or a bad implemented feature.
Looking at the HTML message source, those words are not inserted as they are, Base64 encoding transform your HTML message like:

И this is a test и
>>
=D0=98 this is a test =D0=B8

I don’t think I should enter the encoded form of each character in cPanel.
I do believe developers should update their underlying code and cover this encoding.
Just some staff, please let me know we will have such update in our future?
No any estimated date needed, just if we will get this covered or we should insert each encoded char as well?

Bug in forum as well:
Can't Edit original post, got error: Your message contains elements that seems like kina SPAM!!!
If so, why can make a new post?!

 

[Infopro]

Bug in forum...

This has to do with you being a new forum user. Editing of posts is not permitted by new users.

As for your filter, you might want to post a screenshot (upload it to a post, do not link to an image please) or simply post it here, within code tags, for review, to see whats breaking it.

 

[omidsolo]

Hello and thanks for response
1. So the error being shown is misleading caused me to think it's a bug!
2. Snapshot attached, but as I told, in HTML Base64 encoded email messages these characters are encoded in message and that's why it does not find them, in your code, you should not just check:
If Message.Body.Contains(varMsgBody) = True Then Fail with Message
But you should also check for the encoded varMsgBody as well.
Cause for instance: И is converted to =D0=98 and и to =D0=B8 etc...
I don't know Perl (which I think cPanel is written in) so can't help with code snippet, but it's pretty simple.
Thanks anyway.

 

[cPanelMichael]

Hello @omidsolo,

Can you use the "Filter Test" option in cPanel >> Global Email Filters and upload a screenshot of the results?

Thank you.

 

[omidsolo]

Hi dear Mike,
In filter test it works as expected, because it is Plain Text email message.
In real world most of people use HTML email message with Base64 encoding format.
So as I said for example, Russian Й character inside the source of an HTML email is NOT entered as is.
Й is converted to =D0=99 in HTML.
So if we have Й in our HTML body, your code search in the HTML source and cannot find it because it simply use the InStr (in Perl should be String > Index) to search for Й in the body source.
That's not wrong, but just not enough, in your source you should both:
1. Check Й for example, in Body for example, you already do it
2. (Add) And also check if the encoded form of Й which is =D0=99 also exists in the message source?
The lack of second check break many of user filters, in many languages/characters in HTML Body check filters.
I'm sure that's simple. And snapshot is attached

 

[omidsolo]

Please replace Base64 with Quote Printable, my bad, Base64 is usually used on attachments only, not body.

 

[omidsolo]

And please enable Edit Post, or edit the misleading error it is showing!

 

[Infopro]

And please enable Edit Post, or edit the misleading error it is showing!

Please provide more details of what you need edited. After 24 hours you're unable to edit posts, but we'll be happy to help with that if needed.

 

[omidsolo]

Thanks, that was just an advise, for example, I posted a reply, then wanted to add something else, and had to do it in a new post.
If the policy is to disable edits, I understand, but the error is shows in misleading: Error says that your post triggers SPAM filter!
Then user starts an endless edit to make it possible to post!
Either remove edit button, or show a proper error message.
That was just a suggestion to improve the forum, I don't need edits.
But thanks for the kind response.

 

[cPanelMichael]

Hello @omidsolo,

It would help our investigation if we could access the affected system and review the live filter. Can you ask your hosting provider to open a support ticket with us on your behalf so we can take a closer look? You can ask them for the ticket number they receive from us and post it here once they let you know what it is.

Thank you.

 

[omidsolo]

I already asked them before posting here, and they confirmed this is by design of cPanel so no one can do something about it.
I'm also a .NET developer since 2003 and most of my work was on email systems, so I'm sure about it.
Your filter system is simply like this:
When I select: If Body Contains "Й" (for example) Fail with message
In your code you simply check:

varMsgBody is the variable holding the whole email message body
Й is the Russian character I entered in the filter list (among with some others)

If varMsgBody.Contains("Й") = True Then
'Fail with message
End If

While you should also:

If varMsgBody.Contains("Й") = True Or varMsgBody.Contains(Uri.EscapeUriString("Й")) = True Then
'Fail with message
End If

Because in the inner message source Й is converted to =D0=99 so you should check both for the entered character and it's HTML encoded form so your code logic works in all conditions with all characters.
Please send an email to my address registered on my profile and I allow you to access my cPanel and make a test account, then send an HTML email to it containing my blocked characters and you see that the filter does not work as expected.

The reason this section seems important is that we are getting many many SPAM everyday in Chinese/Russian and we prefer not to receive any email with those characters in Body.
But this filter while is excellent but is not working as expected.

*** Please note that Uri.EscapeUriString("Й") in .NET converts Й to =D0=99
Unfortunately I don't know the equivalent in Perl

 

[cPanelMichael]

Hello @omidsolo,

Thank you for providing us with additional information.

You are correct about the Base64-encoded content. Exim will not automatically decode the Base64-encoded content during the filter rule processing with HTML emails. You can find some additional discussion of this topic on the following thread:

Block custom attachments EXIM issue

As an alternative, you can install and use the SpamAssassin TextCat plugin to filter out emails using specific languages. You can read more about this plugin at:

Mail::SpamAssassin::Plugin::TextCat - TextCat language guesser

To install it, ask your hosting provider to add the following line to the /etc/mail/spamassassin/init.pre file:

loadplugin Mail::SpamAssassin::Plugin::TextCat

Your provider will also need to restart SpamAssassin with the following command after adding the above entry:

/scripts/restartsrv_spamd

Once this plugin is installed, you can add and customize the "ok_languages" SpamAssassin configuration option (documentation on this option is found on the TextCat plugin link above) to your account's /home/username/.spamassassin/user_prefs file.

Let me know if this helps.

Thank you.

 

[omidsolo]

Please note that usually Base64 is used on attachments, which are not my concern here.
I am talking about Body based filtering, that one is encoded as Quoted Printable, for example we set a new global rule:
If Body Contains "Й" Fail with message
This will not work even if you fill the body with hundreds of "Й" characters
Because in HTML email "Й" is converted to =D0=99
So the filter just works for ancient plain text emails, not HTML ones!
I also told the very quick way to fix it, just add one condition to the IF, check for its encoded form as well.

If varMsgBody.Contains("Й") = True Then
'Fail with message
End If

update to:

If varMsgBody.Contains("Й") = True Or varMsgBody.Contains(Uri.EscapeUriString("Й")) = True Then
'Fail with message
End If

That's as easy as ABC to add support for it in a future update, just please forward my whole messages to a real person who is a developer in charge.
-
I will try that other method by then

 

[omidsolo]

OK I asked my hosting provider to install TextCat plugin as suggested, after checking with your support.
I was almost sure it will not work that way, the only thing in this world I'm sure of, are email systems.

There is no certain way of detecting an email language but "language" or "encoding" tag!
Which is useless most of the times, since a Russian email can be UTF-8!
Even if you have some other algorithm, it is not working, my email body was all Russian characters and delivered perfectly.

I even did provide code samples to show your code logic fault but no one cares about the bug or code malfunction!

 

[cPanelMichael]

Hello @omidsolo,

Can you use the Defects link to report this directly? You can post the ticket ID that's assigned to your case on this thread after it's opened.

Thank you.

Update: The ticket lead to the creation of the following feature request:

Global Email Filters - Body.Contains.String Bug

 

[omidsolo]

Well thanks, done, 12236283 & 12236375, I hope someone understand me finally, and care about those bugs! End of story

 

[Nikolay Ilchev]

I think it's a bug the way strings are being bypassed inside HTML bodies in the email.