Under debian, fail2ban’s configuration is in /etc/fail2ban/. In the filter.d directory add the following file and name it apache-modsec.conf.
# Option: failregex
# Notes.: regex to match the password failure messages in the logfile. The
# host must be matched by a group named "host". The tag "<HOST>" can
# be used for standard IP/hostname matching.
# Values: TEX
failregex = [client <HOST>] mod_security: Access denied with code 500
# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
Enable this filter by adding the followin to jail.local:
enabled = true
port = http
filter = apache-modsec
logpath = /var/log/apache*/*error.log
maxretry = 4
Restart fail2ban, next time a spammer gets blocked by modsecurity 4 times, it will be recorded in /var/log/fail2ban.log
2007-02-07 11:52:45,024 fail2ban.actions: WARNING [apache-modsec] Ban 18.104.22.168