Looking to stop microsoft azure bots

jeffschips

Well-Known Member
Jun 5, 2016
261
33
78
new york
cPanel Access Level
Root Administrator
Trying to stop the onslaught of microsoft azure bots. I have a script that stops the amazon ones in csf but can't find one for the microsoft flavor. Also wondering if there is a third party provider that does this for the microsoft bots integrated into mod_security.

Any advise appreciated.
 

cPRex

Jurassic Moderator
Staff member
Oct 19, 2014
11,793
1,878
363
cPanel Access Level
Root Administrator
Hey hey! It seems there isn't going to be a list of IPs available, as Microsoft says this:

"Bot Framework Services is hosted in Azure data centers world-wide and the list of Azure IPs is constantly changing. That means allow-listing certain IP addresses may work one day and break the next as the Azure IP Addresses change."

but just above that section on the same page they do publish a list of hostnames that are allowed. You could use that list to block traffic instead:


While we don't provide support for CSF, their dyndns tool will likely help take care of this. More details about that functionality can be found here:

 

jeffschips

Well-Known Member
Jun 5, 2016
261
33
78
new york
cPanel Access Level
Root Administrator
So in essence you are suggesting that I take the list of microsoft approved bot hosts and then ask csf to deny bots hiting my server originating from that list of domains, correct?
 

ejsolutions

Well-Known Member
Jan 6, 2013
77
32
68
cPanel Access Level
Root Administrator
Caution: AFAIK csf.dyndns is intended to IGNORE dynamic IP addresses, not to DENY!
I use it to map my home ISP-derived IP, so that I don't inadvertently lock myself out of a server.

( In theory, there's no reason why csf.dyndns couldn't/shouldn't be made available for other purposes, though I suspect this isn't coded in CSF. You'd be best to confirm with WayToTheWeb and perhaps place a feature request.)
 

ejsolutions

Well-Known Member
Jan 6, 2013
77
32
68
cPanel Access Level
Root Administrator
This looks to be a better resource and may inspire me to 'play' a little:
It's updated weekly and is presented as a json file (yuk!). I have a downloaded script that parses other json lists, for different services (Hetzner, AWS etc.) based on ASN (IIRC). It might be adaptable..

[Much later and multi-edits]
This appears to work, after much routing around and piecing together various snippets.

Step 1. Edit /etc/csf/csf.blocklists and add to the bottom, changing the FQDN hostname):
AZUREIP|86400|0|http://your.whmserver.tld/azure-ip.txt

Step 2. Create, for example, /root/azure-ip.sh with the following:
Code:
#!/bin/bash
# Produce http (txt) lists for other servers
dir=/usr/local/apache/htdocs
# Comment out existing IPs to allow new list retrieval (just to be sure its not blocked)
sed -i 's/AZUREIP/#AZUREIP/g' /etc/csf/csf.blocklists
/usr/sbin/csf -r
mv $dir/ip-ranges.json $dir/ip-ranges.json.old
# Find the updated list
newlist=`curl -sS https://www.microsoft.com/en-us/download/confirmation.aspx?id=56519 | egrep -o 'https://download.*?\.json' | grep -v meta | uniq`
# Grab the list
wget -O $dir/ip-ranges.json $newlist
# Strip off the crud, to leave CIDR IPs
grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}/[0-9]{1,2}" $dir/ip-ranges.json > $dir/azure-ip.txt
# Reactivate IP ranges, as a group in the blocklists
sed -i 's/#AZUREIP/AZUREIP/g' /etc/csf/csf.blocklists
# Activate the new list
/usr/sbin/csf -r
Step 3. Finally, run "crontab -e" and add a weekly task, for example:
28 1 * * 1 /root/azure-ip.sh > /dev/null 2>&1

You can also access this generated list from other WHM servers by adding the same entry to /etc/csf/csf.blocklists

Hope that's useful.
 
Last edited:
  • Like
Reactions: cPRex

jeffschips

Well-Known Member
Jun 5, 2016
261
33
78
new york
cPanel Access Level
Root Administrator
Hi ejsolutions that script is very useful. It seems to break right after here though:

# Add in new IP ranges, as a group in the blocklists
sed -i 's/#AZUREIP/AZUREIP/g' /etc/csf/csf.blocklists
# Activate the new list
/usr/sbin/csf -r

It's not inserting the azure-ip.txt into the csf.blocklists

I can confirm that azure-ip.txt is being created though in /usr/local/apache/htdocs

Maybe something to do with paths?
 

jeffschips

Well-Known Member
Jun 5, 2016
261
33
78
new york
cPanel Access Level
Root Administrator
I also noticed a warning in the csf.blocklists.new file:

# If you want to redownload a blocklist you must first delete
# /var/lib/csf/csf.block.NAME and then restart csf and then lfd

Wonder if that is important. . .
 

ejsolutions

Well-Known Member
Jan 6, 2013
77
32
68
cPanel Access Level
Root Administrator
It's not inserting the azure-ip.txt into the csf.blocklists
What it's supposed to do, is remove the prefixed # (comment) from the line in csf.blocklists
Look carefully at your shell script in case your copy & paste got screwed up - quotation marks get changed by many editors, for example.

Did you remember the first step?
Edit /etc/csf/csf.blocklists and add to the bottom, changing the FQDN hostname):
AZUREIP|86400|0|http://your.whmserver.tld/azure-ip.txt
I've never had to mess around with csf.blocklists.new - didn't know it existed!

The bottom of my csf.blocklists currently looks like this..
 
Last edited:

ejsolutions

Well-Known Member
Jan 6, 2013
77
32
68
cPanel Access Level
Root Administrator
I should've stated the obvious?:
Change your.whmserver.tld and/or mywhm.server.com to match your own server hostname.
To test operation, manually run /root/azure-ip.sh to initially update the list.
 

ejsolutions

Well-Known Member
Jan 6, 2013
77
32
68
cPanel Access Level
Root Administrator
Note that the script works as expected, based on their (MS) own publicly available Azure IPs.
If there's a list of IPs that outlook.com email servers use, then that'd be handy - would I whitelist 'em though? :rolleyes:

This problem is not exclusive to Microsoft; many service providers, such as CSF, Let'sEncrypt, etc. use mirrors on well-known port scanning/hacking attempt networks, such as Hetzner, Contabo and Digital Ocean. Bad decisions, IMHO. I do wonder how much bandwidth usage and processing power would be reduced, if these rogue network packets were blocked at source, as they should be. The instigators could get jailed too! /rant.
 
Last edited: