block userAgent on server, with mod_security?

Instead of using modsec to achieve this, you could use bad-bot-blocker in a global .htaccess file.

An example of b-b-b can be found here.

 

Do you have the correct permissions on that file? What does the apache error_log file say?

644 should suffice for /home/.htaccess

 

These are the rules I use for that attack:

Code:

#Joomla com_jce exploit
SecRule HTTP_User-Agent "BOT for JCE" "deny,status:500,id:5000218,msg:'Joomla com_jce code exec'"

#Joomla com_jce exploit
SecRule REQUEST_URI "/images/stories/.+\.php" "deny,status:500,id:5000219,msg:'Joomla com_jce code exec'"

The first blocks the user agent. That exploit puts PHP files into site.com/images/stories/something.php if it is successful, so the 2nd rule blocks access to those in case they change user agent.

Even with the .htaccess or this first rule, you should still use the 2nd rule. Changing user agents is very simple.