Sounds like a great idea. But doesn't this mean you need to keep spamming the various whois servers, which might conceivably be against their TOS? Also, don't the whois servers eventually ratelimit or block you at your end?
A WHOIS only occurs once per domain (google.com had a lookup performed on it ONCE, every lookup after that is internally cached). A this point I have 1,812,859 cached domain lookups (Once whitelisted, I have no reason to ever lookup that domain again). At any given instant, there is about on average 2,500 "fresh domains" actively blacklisted (also cached).
From there, there's other logic to prevent and ratelimit lookups for WHOIS lookups that fail (be it the domain is not registered or some actual failure) specifically to try and prevent abuse of a given TLD's whois service. Failed lookups simply return a negative response (not blacklisted) to the MTA to err on the side of allowing mail in. There's also logic to shift the blacklisted item into the whitelisted cache after its 5 day expiry time. Literally a registered domain is only ever looked up once.
The amount of actual WHOIS lookups performed per-TLD per-day is actually fairly reasonable (so far). But you are correct, all it takes is a TLD organization to disagree and (either by request or by blocking) cause the lookups to cease for their TLD and render the DNSBL useless for their TLD.
It's been operating since May or so without issue, but that's just anecdotal. The risk definitely is there.
From there, there's other logic to prevent and ratelimit lookups for WHOIS lookups that fail (be it the domain is not registered or some actual failure) specifically to try and prevent abuse of a given TLD's whois service. Failed lookups simply return a negative response (not blacklisted) to the MTA to err on the side of allowing mail in. There's also logic to shift the blacklisted item into the whitelisted cache after its 5 day expiry time. Literally a registered domain is only ever looked up once.
The amount of actual WHOIS lookups performed per-TLD per-day is actually fairly reasonable (so far). But you are correct, all it takes is a TLD organization to disagree and (either by request or by blocking) cause the lookups to cease for their TLD and render the DNSBL useless for their TLD.
It's been operating since May or so without issue, but that's just anecdotal. The risk definitely is there.
Nice work BOates!
I read through the thread and figured it was worth a shot. I put this on one of my smaller boxes that doesn't have a ton of mailboxes and within 10 minutes we caught the below domains.
What's nice is based on the sender name before the new domain you can easily see that this is simply spam.
- Removed - No Need For This Here -
I read through the thread and figured it was worth a shot. I put this on one of my smaller boxes that doesn't have a ton of mailboxes and within 10 minutes we caught the below domains.
What's nice is based on the sender name before the new domain you can easily see that this is simply spam.
- Removed - No Need For This Here -
Last edited by a moderator:
Thanks for the reply. One possibility I can think of is that a non-spamming, whitelisted domain could expire, and is then subsequently registered as a temporary throwaway by a spammer. To counter this would require at most one additional valid whois check per year per domain, but with millions of domains to recheck, it would quickly add up and possibly greatly increase the risk of being blocked.
Yep, that's a loophole in my current method. However, at least at the moment, I have not seen this behavior and the added queries to check against this loophole would make the DNSBL more invasive to WHOIS servers like you say.
One way I justified that it's not as important to re-check expired and re-registered domains is a similar reason why spammers seem to want "fresh" domains. All this spam that is being sent out is, eventually, over the course of a few days/a week, resulting in many URIBLS and other domain reputation based lists catching them. Even still whitelisted against my DNSBL, it would be more likely that some other block list WOULD have them blacklisted still. Similar to how people inherit IP reputation of the previous owner, I imagine/hope that the domains would retain their poor reputation and hopefully be considered useless to spammers.
You are correct on the potential for legitimate/good reputation domains expiring and then being abused by spammers. At the outset, I think my response to that would be that attempting to be that "aggressive" in response with the added WHOIS queries needed would be beyond the scope of this DNSBL. Spamming WHOIS servers is not something I want to intentionally do.
I could be wrong. But, you never know. If this DNSBL becomes reasonably popular, circumventing it may be eventually worth the effort.
One way I justified that it's not as important to re-check expired and re-registered domains is a similar reason why spammers seem to want "fresh" domains. All this spam that is being sent out is, eventually, over the course of a few days/a week, resulting in many URIBLS and other domain reputation based lists catching them. Even still whitelisted against my DNSBL, it would be more likely that some other block list WOULD have them blacklisted still. Similar to how people inherit IP reputation of the previous owner, I imagine/hope that the domains would retain their poor reputation and hopefully be considered useless to spammers.
You are correct on the potential for legitimate/good reputation domains expiring and then being abused by spammers. At the outset, I think my response to that would be that attempting to be that "aggressive" in response with the added WHOIS queries needed would be beyond the scope of this DNSBL. Spamming WHOIS servers is not something I want to intentionally do.
I could be wrong. But, you never know. If this DNSBL becomes reasonably popular, circumventing it may be eventually worth the effort.
Last edited:
What's the creation date on the WHOIS data for the domains you're seeing get through, and what's the TLD in use -- are they all the same TLD? If you want to PM me the domains I'll check into it.
Just looking at the first spam that just came through, it's example.eu which was registered yesterday.
Last edited by a moderator:
(Edit) That's why, the public WHOIS server for that registrar does not report any data over the normal WHOIS service. They redirect you to go to their proprietary web page and do a lookup there.
My DNSBL requires that the creation date be viewable via the standard WHOIS service lookup. So any domains registered by that particular registrar are not able to be filtered by my DNSBL.
My DNSBL requires that the creation date be viewable via the standard WHOIS service lookup. So any domains registered by that particular registrar are not able to be filtered by my DNSBL.
Last edited by a moderator:
I am testing this on 2 servers before making the change to all my servers I have.
I will you all informed after a couple days of testing on how this goes.
I will you all informed after a couple days of testing on how this goes.
I received tens of messages today from the .LOAN TLD, which looks like it's not returning results from [Removed]
Few examples that were registered today:
[Removed]
The public WHOIS service brings back the correct creation date.
Few examples that were registered today:
[Removed]
The public WHOIS service brings back the correct creation date.
Last edited by a moderator:
BOates, I've been using this for over a year now, however the last week I have been getting tons of spam. Is the system down?
Looks like ".loan" is among some newer TLDs that weren't manually configured in the system with its proper WHOIS server. These should now be trapped by the DNSBL; just added it in now. I'm going to have to go perusing for new TLDs again and add any missing ones' whois servers.
Nope, the system has been operating essentially 24/7 since inception. Any trends to the mail being let through? PM me any particular examples or list the TLDs that seem commonly let through and I'll check on it. All the spam getting through is coming from domains registered within the last few days?
Thanks,
Today's TLD seems to be .download
Yesterday's was a .com, however registered 27th Jan, so I guess it just wasn't new enough.
Today's TLD seems to be .download
Yesterday's was a .com, however registered 27th Jan, so I guess it just wasn't new enough.
Yep this list is still up and running. Still effective and I've still been using it for my personal mail/server.
Usage of CMM shouldn't affect or change the implementation of the list. You should be able to follow the original instructions all the same.
It seems that for me something is not working, I cannot seem to see anything from it in logs, and I am unable to get any results from dig +short test.fresh.dieinafire.com . I would like a tutorial to add this via CMM.
