mod_evasive issues

[manager23]

I was facing a DDOS attack and installed mod_evasive. Ran into a few issues, some related to configuration (so I'll indicate how I've solved them, for others that face the same issue), some I think are bugs in mod_evasive.

ISSUE 1 (bug?):
Setting DOSBlockingPeriod in /etc/apache2/conf.d/300-mod_evasive.conf doesn't have any effect (yes, I did run "service httpd reload"). I wanted to set it to 300 (seconds), but it still uses the default setting 10 (seconds). It's not defined anywhere else (grepped all files in the /etc/apache2 tree).

ISSUE 2 (configuration):
If you leave DOSBlockingHttpResponse set to 403 and you have ErrorDocument set for 403 responses which require a 403.shtml file in public_html... then you get a 404 error if that file is missing. Changing DOSBlockingHttpResponse to 429 solves that, because (at least in my config) it doesn't have an ErrorDocument attached to it, so it just works. Plus it's a much clearer error message to distinguish it from actual permission problems that might also result in a 403.

ISSUE 3 (configuration):
If setting DOSEmailNotify doesn't result in mails getting sent, in WHM go to "Tweak settings", search for "nobody" and set the "Prevent “nobody” from sending mail" to "off".

ISSUE 4 (bug or feature?):
I only seem to get email notifications when triggering DOSSiteCount, but not when triggering DOSPageCount.
edit: found the cause, it's not related to the type of notification: #931813 - libapache2-mod-evasive: failure to remove dos-* log files breaks further logging and notifications - Debian Bug report logs
Apparently it only sends an email if the IPnr (i.e. 1.2.3.4) isn't listed as a file (i.e. dos-1.2.3.4) in /var/log/apache2/mod_evasive yet (DOSLogDir). This avoids getting emails about the same IPnr repeatedly, but you may want to clean up old files in that directory, for example by adding a hourly cronjob that cleans up the "dos-*" files older than one day. That way, if the same IP gets blocked again, you will at least get a new notification once a day.

0 * * * * /usr/bin/find /var/log/apache2/mod_evasive -ctime +1 -type f | /usr/bin/xargs -r rm

ISSUE 5 (bug?):
The notifications that I'm getting had an empty subject and what was supposed to be in headers (subject / to) ended up in the body of the email.
A very dirty way to fix that is:

sed -i 's/mail %s/mail -t/' /usr/lib64/apache2/modules/mod_evasive24.so

Originally, it calls the /bin/mail program with the recipient email addres as argument (%s), but if you leave out the argument and instead supply the option "-t" it parses the supplied headers for the recipient address. Solved the issue for me, although it'll probably be undone if there's ever an update of the mod_evasive package installed.
edit: see also #776635 - libapache2-mod-evasive: mails are badly formated - Debian Bug report logs

ISSUE 6 (configuration):
If you have multiple WHM servers, you get notification mails that all come from Nobody. Although you can see the actual emailadress ([email protected]), you can change the name in /etc/passwd. 5th parameter on the line that starts with "nobody" is the name used. Changed that from "Nobody" to "Nobody - my.server.tld" so I could distinguish the mails better in my mailbox.

ISSUE 7 (wishlist):
If issue 4 gets fixed, it would be really nice if the notification mail didn't just show the IP addres, but also the URL that was requested too many times. Or even include it in both DOSSite and DOSPage notifications, but in that case indicate which type of notification Site/Page it is. Otherwise one has to search all access logs to figure out which website was attacked (I was hoping for some kind of log entry somewhere in /var/log, but couldn't find it).

ISSUE 8: Wordpress sites (configuration):
Probably happens with other websites as well, but the default DOSSiteCount 100 setting immediately got me blocked when visiting one of the hosted wordpress website, because one page consisted of way more than 100 elements which got loaded within the 2 seconds set in DOSSiteInterval and thus triggerd mod_evasive. I've bumped that setting to 500.
The DDOS I faced had a few hundred IPnrs each requesting the same URL, but at a relatively slow rate (once per 2-5 seconds), so the default setting DosPageCount 4 and DosPageInterval 2 (which requires over 2 requests per seconds to start blocking) didn't have any effect. So I raised that to blocking when exceeding 3 requests for the same URL per 20 seconds. However, if you click through the Wordpress backend, on most pages it loads the exact same JS (and other) files, so clicking 4 times in the backend resulted in blocking. Raised that to 15 identical requests per 60 seconds, which may still require extra tuning.

I'm hoping for a mod_evasive package update for the issues marked bug/wishlist (if there's another way to fix it, that's fine too).

--Ruud

 

[bellwood]

You're best posting here (I think):

GitHub - CpanelInc/ea-apache24-mod_evasive

Contribute to CpanelInc/ea-apache24-mod_evasive development by creating an account on GitHub.

github.com

...not that anyone from cPanel actually appears to be monitoring that repo.

I get the sense the cPanel simply forked shivaas mod_evasive distro and then mothballed it. It would be nice to see some improvements and I think there's some to be added from other maintainers, but, with no sense of communication on the repo, I'm sure many feel it's not worth the effort to submit pulls if they'll go upon deaf ears.

 

[cPRex]

@manager23 - thanks for taking the time to write this up. I'll do some additional testing on my end tomorrow and send you some more details then.

 

[cPRex]

@manager23 - we've had some things come up so I'm not sure if I'll get to this today. I will definitely test this and report back!

 

[cPRex]

I've talked with our team and confirmed that cPanel doesn't make any customizations to the mod_evasive module, so if there are issues with the tool itself it would best be reported to the creator here:

GitHub - jzdziarski/mod_evasive: Apache mod_evasive module

Apache mod_evasive module. Contribute to jzdziarski/mod_evasive development by creating an account on GitHub.

github.com

I did note in their documentation they say this:

DOSBlockingPeriod
-----------------

The blocking period is the amount of time (in seconds) that a client will be
blocked for if they are added to the blocking list.  During this time, all
subsequent requests from the client will result in a 403 (Forbidden) and
the timer being reset (e.g. another 10 seconds).  Since the timer is reset
for every subsequent request, it is not necessary to have a long blocking
period; in the event of a DoS attack, this timer will keep getting reset.

so that would explain issue #1 that you're experiencing - the timer gets reset on each new connection, so if you're actively seeing a DoS against your system, it will never be able to reach 300 seconds as new connections from the same address would reset the timer.

For the other issues mentioned, it would be best to reach out to the developer of the plugin directly as cPanel doesn't have a way to resolve this without making our own custom version of the mod_evasive tool, which isn't something we plan to pursue at this time.

 

[manager23]

@cPRex: About DOSBlockingPeriod: I don't think it explains #1. I'm testing this by first triggering a block via a quick series of requests. Then wait 15 seconds before doing another request. If I have set the DOSBlockingPeriod to 30, I would expect to still be blocked, but the block is removed after 10 seconds of inactivity, regardless of what I specify for DOSBlockingPeriod.

But I'll report it to the upstream maintainer. Should I report back here if upstream code is updated or is that checked automatically somehow?

edit (wanted to edit topic start, but probably too late):
ISSUE 9: mod_evasive vs mod_remoteip
If you have both installed are using (for example) Cloudflare and want to whitelist the original visitor IPnr, you'll find that while mod_remoteip can retrieve the original visitor IPnr and it gets stored in access logs just fine, mod_evasive unfortunately ignores what mod_remoteip does and still uses the Cloudflare IPnrs.
If you then try to whitelist the netblocks on https://www.cloudflare.com/ips-v4 and don't read mod_evasive documentation, you'll probably try whitelisting
162.158.0.0/15 via 162.158.* and 162.159.*. Doesn't work, because mod_evasive only allows the last octet to be a wildcard, so you have to convert the CIDR notation into a series of class C networks. I used the sipcalc tool (Ubuntu Manpage: sipcalc - IP subnet calculator) to do it like this, because you end up with a list of 6000+ netblocks, so it's not something you want to do manually:

curl https://www.cloudflare.com/ips-v4 | \
  xargs -n1 sipcalc -s /24 | \
  grep Network | \
  cut -d- -f2 | \
  cut -d' ' -f2 | \
  sed -e 's/\.0$/.*/' -e 's/^/DOSWhitelist /' \
  > /etc/apache2/mod_evasive.cloudflare.ipv4.conf

And then add this to the mod_evasive config file followed by a reload of the apache config (service httpd reload):

Include /etc/apache2/mod_evasive.cloudflare.ipv4.conf

You'd have to do this periodically, in case Cloudflare adds more netblocks.

edit2 (adding to issue 9):
In Cloudflare DNS, you also have to make sure the IPnr of the cPanel server is only listed via IPv4 (A) and not IPv6 (AAAA), otherwise you'd have to whitelist IPv6 netblocks as well. Didn't find documentation on how to whitelist IPv6 addresses. Looking at the code, it seems to be targeted only at IPv4 addresses... and seems I misread the documentations. For IPv4, instead of 162.158.* (which doesn't give an error, but doesn't work either) it seems that 162.158.*.* would work, so you can use wildcards like 1.*.*.*, 1.2.*.* and 1.2.3.* to match class A, B and C netblocks respectively. You'd still have to convert from the CIDR notation cloudflare uses to a series of class B and C networks (which results in a smaller set than just generating class C netblocks). Leaving that as an exercise to the reader

 

[cPRex]

Feel free to post back here if you do hear from the upstream crew about any changes!

 

[Steve Kemp]

I just noticed the issue about the nobody@ email issues with my own mod_evasive installs but i found that the fix suggested:

ISSUE 5 (bug?):
The notifications that I'm getting had an empty subject and what was supposed to be in headers (subject / to) ended up in the body of the email.
A very dirty way to fix that is:

sed -i 's/mail %s/mail -t/' /usr/lib64/apache2/modules/mod_evasive24.so

Originally, it calls the /bin/mail program with the recipient email addres as argument (%s), but if you leave out the argument and instead supply the option "-t" it parses the supplied headers for the recipient address. Solved the issue for me, although it'll probably be undone if there's ever an update of the mod_evasive package installed.
edit: see also #776635 - libapache2-mod-evasive: mails are badly formated - Debian Bug report logs

doesn't seem to work, the subject is still in the email content and the subject line is still blank.

is there any other way of resolving this?