Is there a reason that cPHulk Brute Force Protection is not able to handle this for you? If the user actually attempts to authenticate, then cPHulk should work under dovecot to blacklist the IP.
Now, if the user simply tries a real DoS where they do not try to authenticate but simply hit the service and freeze up the connections, then cPHulk will not stem that type of attack. It is relatively easy to add firewall rules to stop a set number of connections beyond a certain amount into iptables directly such as the following example:
Code:
/sbin/iptables -A INPUT -i eth0 -p tcp --dport 110 -m state --state NEW -m recent --set --name POP3
/sbin/iptables -A INPUT -i eth0 -p tcp --dport 110 -m state --state NEW -m recent --update --seconds 60 --hitcount 8 --rttl --name POP3 -j DROP
You would then do another for IMAP services with the --dport listed as that port number and change the --name portion to IMAP. This would block any new connections that hit the machine more than 8 times in 60 seconds.