Looking to stop microsoft azure bots

[jeffschips]

Trying to stop the onslaught of microsoft azure bots. I have a script that stops the amazon ones in csf but can't find one for the microsoft flavor. Also wondering if there is a third party provider that does this for the microsoft bots integrated into mod_security.

Any advise appreciated.

 

[cPRex]

Hey hey! It seems there isn't going to be a list of IPs available, as Microsoft says this:

"Bot Framework Services is hosted in Azure data centers world-wide and the list of Azure IPs is constantly changing. That means allow-listing certain IP addresses may work one day and break the next as the Azure IP Addresses change."

but just above that section on the same page they do publish a list of hostnames that are allowed. You could use that list to block traffic instead:

Bot Framework Security and Privacy Frequently Asked Questions - Bot Service

Answers to common security and privacy frequently asked questions.

docs.microsoft.com

While we don't provide support for CSF, their dyndns tool will likely help take care of this. More details about that functionality can be found here:

https://www.interserver.net/tips/kb/block-domains-having-dynamic-ips-using-csf/

 

[jeffschips]

So in essence you are suggesting that I take the list of microsoft approved bot hosts and then ask csf to deny bots hiting my server originating from that list of domains, correct?

 

[ejsolutions]

Caution: AFAIK csf.dyndns is intended to IGNORE dynamic IP addresses, not to DENY!
I use it to map my home ISP-derived IP, so that I don't inadvertently lock myself out of a server.

( In theory, there's no reason why csf.dyndns couldn't/shouldn't be made available for other purposes, though I suspect this isn't coded in CSF. You'd be best to confirm with WayToTheWeb and perhaps place a feature request.)

 

[ejsolutions]

This looks to be a better resource and may inspire me to 'play' a little:

Download Azure IP Ranges and Service Tags – Public Cloud from Official Microsoft Download Center

www.microsoft.com

It's updated weekly and is presented as a json file (yuk!). I have a downloaded script that parses other json lists, for different services (Hetzner, AWS etc.) based on ASN (IIRC). It might be adaptable..

[Much later and multi-edits]
This appears to work, after much routing around and piecing together various snippets.

Step 1. Edit /etc/csf/csf.blocklists and add to the bottom, changing the FQDN hostname):
AZUREIP|86400|0|http://your.whmserver.tld/azure-ip.txt

Step 2. Create, for example, /root/azure-ip.sh with the following:

#!/bin/bash
# Produce http (txt) lists for other servers
dir=/usr/local/apache/htdocs
# Comment out existing IPs to allow new list retrieval (just to be sure its not blocked)
sed -i 's/AZUREIP/#AZUREIP/g' /etc/csf/csf.blocklists
/usr/sbin/csf -r
mv $dir/ip-ranges.json $dir/ip-ranges.json.old
# Find the updated list
newlist=`curl -sS https://www.microsoft.com/en-us/download/confirmation.aspx?id=56519 | egrep -o 'https://download.*?\.json' | grep -v meta | uniq`
# Grab the list
wget -O $dir/ip-ranges.json $newlist
# Strip off the crud, to leave CIDR IPs
grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}/[0-9]{1,2}" $dir/ip-ranges.json > $dir/azure-ip.txt
# Reactivate IP ranges, as a group in the blocklists
sed -i 's/#AZUREIP/AZUREIP/g' /etc/csf/csf.blocklists
# Activate the new list
/usr/sbin/csf -r

Step 3. Finally, run "crontab -e" and add a weekly task, for example:
28 1 * * 1 /root/azure-ip.sh > /dev/null 2>&1

You can also access this generated list from other WHM servers by adding the same entry to /etc/csf/csf.blocklists

Hope that's useful.

 

[jeffschips]

Hi ejsolutions that script is very useful. It seems to break right after here though:

# Add in new IP ranges, as a group in the blocklists
sed -i 's/#AZUREIP/AZUREIP/g' /etc/csf/csf.blocklists
# Activate the new list
/usr/sbin/csf -r

It's not inserting the azure-ip.txt into the csf.blocklists

I can confirm that azure-ip.txt is being created though in /usr/local/apache/htdocs

Maybe something to do with paths?

 

[jeffschips]

I also noticed a warning in the csf.blocklists.new file:

# If you want to redownload a blocklist you must first delete
# /var/lib/csf/csf.block.NAME and then restart csf and then lfd

Wonder if that is important. . .

 

[ejsolutions]

It's not inserting the azure-ip.txt into the csf.blocklists

What it's supposed to do, is remove the prefixed # (comment) from the line in csf.blocklists
Look carefully at your shell script in case your copy & paste got screwed up - quotation marks get changed by many editors, for example.

Did you remember the first step?

Edit /etc/csf/csf.blocklists and add to the bottom, changing the FQDN hostname):
AZUREIP|86400|0|http://your.whmserver.tld/azure-ip.txt

I've never had to mess around with csf.blocklists.new - didn't know it existed!

The bottom of my csf.blocklists currently looks like this..

# Digital Ocean
# DOCEAN|86400|0|https://asn.ipinfo.app/api/text/list/AS14061
# Contabo
# CONTABO|86400|0|https://asn.ipinfo.app/api/text/list/51167
# Hetzner
# HETZNER|86400|0|https://asn.ipinfo.app/api/text/list/AS24940

# AWSIP|86400|0|http://mywhm.server.com/aws-ip.txt
SHODANIP|86400|0|http://mywhmserver.com/shodan-ip.txt
AZUREIP|86400|0|http://mywhm.server.com/azure-ip.txt

 

[jeffschips]

Hello @ejsolutions

Your suggestions look spot-on. I'll implement them and report back.

Thanks so much.

 

[ejsolutions]

I should've stated the obvious?:
Change your.whmserver.tld and/or mywhm.server.com to match your own server hostname.
To test operation, manually run /root/azure-ip.sh to initially update the list.

 

[jeffschips]

Thank you @ejsolutions this works!

Misison accomplished!

SOLVED

 

[ejsolutions]

@jeffschips ( @cPRex )
** WARNING **
Utilising this method will block email to/from outlook.com. MS in their eternal wisdom appears to use their own customer Azure network for core functions - eejits!

 

[jeffschips]

Yikes-er-roni!

 

[ejsolutions]

Note that the script works as expected, based on their (MS) own publicly available Azure IPs.
If there's a list of IPs that outlook.com email servers use, then that'd be handy - would I whitelist 'em though?

This problem is not exclusive to Microsoft; many service providers, such as CSF, Let'sEncrypt, etc. use mirrors on well-known port scanning/hacking attempt networks, such as Hetzner, Contabo and Digital Ocean. Bad decisions, IMHO. I do wonder how much bandwidth usage and processing power would be reduced, if these rogue network packets were blocked at source, as they should be. The instigators could get jailed too! /rant.

 

[xpy-xpy]

Found this thread after trying to stop a bot attack coming from AWS IPs. Can confirm that it is not just Azure. Blocking AWS ranges turned out to be a bad idea too. LiteSpeed plugin was the first thing to break immediately, before I stopped the experiment.