Move action.d/mail-whois-common.conf into fail2ban-server

ver. 0.10.5 (2020/01/10) - deserve-more-respect-a-jedis-weapon-must

Yes, Hrrrm...

Fixes

• [compatibility] systemd backend: default flags changed to SYSTEM_ONLY(4), fixed in gh-2444 in order to ignore user session files per default, so could prevent "Too many open files" errors on a lot of user sessions (see gh-2392)
• [grave] fixed parsing of multi-line filters (maxlines > 1) together with systemd backend, now systemd-filter replaces newlines in message from systemd journal with \n (otherwise multi-line parsing may be broken, because removal of matched string from multi-line buffer window is confused by such extra new-lines, so they are retained and got matched on every followed message, see gh-2431)
• [stability] prevent race condition - no unban if the bans occur continuously (gh-2410); now an unban-check will happen not later than 10 tickets get banned regardless there are still active bans available (precedence of ban over unban-check is 10 now)
• fixed read of included config-files (.local overwrites options of .conf for config-files included with before/after)
• action.d/abuseipdb.conf: switched to use AbuseIPDB API v2 (gh-2302)
• action.d/badips.py: fixed start of banaction on demand (which may be IP-family related), gh-2390
• action.d/helpers-common.conf: rewritten grep arguments, now options -wF used to match only whole words and fixed string (not as pattern), gh-2298
• filter.d/apache-auth.conf:
• ignore errors from mod_evasive in normal mode (mode-controlled now) (gh-2548);
• extended with option mode - normal (default) and aggressive
• filter.d/sshd.conf:
• matches Bad protocol version identification in ddos and aggressive modes (gh-2404).
• captures Disconnecting ...: Change of username or service not allowed (gh-2239, gh-2279)
• captures Disconnected from ... [preauth], preauth phase only, different handling by extra (with supplied user only) and ddos/aggressive mode (gh-2115, gh-2239, gh-2279)
• filter.d/mysqld-auth.conf:
• MYSQL 8.0.13 compatibility (log-error-verbosity = 3), log-format contains few additional words enclosed in brackets after "[Note]" (gh-2314)
• filter.d/sendmail-reject.conf:
• mode=extra now captures port IDs of TLSMTA and MSA (defaults for ports 465 and 587 on some distros)
• files/fail2ban.service.in: fixed systemd-unit template - missing nftables dependency (gh-2313)
• several action.d/mail*: fixed usage with multiple log files (ultimate fix for gh-976, gh-2341)
• filter.d/sendmail-reject.conf: fixed journal usage for some systems (e. g. CentOS): if only identifier set to sm-mta (no unit sendmail) for some messages (gh-2385)
• filter.d/asterisk.conf: asterisk can log additional timestamp if logs into systemd-journal (regex extended with optional part matching this, gh-2383)
• filter.d/postfix.conf:
  • regexp's accept variable suffix code in status of postfix for precise messages (gh-2442)
  • extended with new postfix filter mode errors to match "too many errors" (gh-2439), also included within modes normal, more (extra and aggressive), since postfix parameter smtpd_hard_error_limit is default 20 (additionally consider maxretry)
• filter.d/named-refused.conf:
  • support BIND 9.11.0 log format (includes an additional field @0xXXX..., gh-2406);
  • prefregex extended, more selective now (denied/NOTAUTH suffix moved from failregex, so no catch-all there anymore)
• filter.d/sendmail-auth.conf, filter.d/sendmail-reject.conf :
• ID in prefix can be longer as 14 characters (gh-2563);
• all filters would accept square brackets around IPv4 addresses also (e. g. monit-filter, gh-2494)
• avoids unhandled exception during flush (gh-2588)
• fixes pass2allow-ftp jail - due to inverted handling, action should prohibit access per default for any IP, therefore reset start on demand parameter for this action (it will be started immediately by repair);
• auto-detection of IPv6 subsystem availability (important for not on-demand actions or jails, like pass2allow);

New Features

• new replacement tags for failregex to match subnets in form of IP-addresses with CIDR mask (gh-2559):
• <CIDR> - helper regex to match CIDR (simple integer form of net-mask);
• <SUBNET> - regex to match sub-net adresses (in form of IP/CIDR, also single IP is matched, so part /CIDR is optional);
• grouped tags (<ADDR>, <HOST>, <SUBNET>) recognize IP addresses enclosed in square brackets
• new failregex-flag tag <F-MLFGAINED> for failregex, signaled that the access to service was gained (ATM used similar to tag <F-NOFAIL>, but it does not add the log-line to matches, gh-2279)
• filters: introduced new configuration parameter logtype (default file for file-backends, and journal for journal-backends, gh-2387); can be also set to rfc5424 to force filters (which include common.conf) to use RFC 5424 conform prefix-line per default (gh-2467);
• for better performance and safety the option logtype can be also used to select short prefix-line for file-backends too for all filters using __prefix_line (common.conf), if message logged only with hostname svc[nnnn] prefix (often the case on several systems):

[jail]
backend = auto
filter = flt[logtype=short]

• filter.d/common.conf: differentiate __prefix_line for file/journal logtype's (speedup and fix parsing of systemd-journal);
• filter.d/traefik-auth.conf: used to ban hosts, that were failed through traefik
• filter.d/znc-adminlog.conf: new filter for ZNC (IRC bouncer); requires the adminlog module to be loaded

Enhancements

• introduced new options: dbmaxmatches (fail2ban.conf) and maxmatches (jail.conf) to contol how many matches per ticket fail2ban can hold in memory and store in database (gh-2402, gh-2118);
• fail2ban.conf: introduced new section [Thread] and option stacksize to configure default size of the stack for threads running in fail2ban (gh-2356), it could be set in fail2ban.local to avoid runtime error "can't start new thread" (see gh-969);
• jail-reader extended (amend to gh-1622): actions support multi-line options now (interpolations containing new-line);
• fail2ban-client: extended to ban/unban multiple tickets (see gh-2351, gh-2349); Syntax:
• fail2ban-client set <jain> banip <ip1> ... <ipN>
• fail2ban-client set <jain> unbanip [--report-absent] <ip1> ... <ipN>
• fail2ban-client: extended with new feature which allows to inform fail2ban about single or multiple attempts (failure) for IP (resp. failure-ID), see gh-2351; Syntax:
• fail2ban-client set <jail> attempt <ip> [<failure-message1> ... <failure-messageN>]
• action.d/nftables.conf:
• isolate fail2ban rules into a dedicated table and chain (gh-2254)
• nftables-allports supports multiple protocols in single rule now
• combined nftables actions to single action nftables:
  • nftables-common is removed (replaced with single action nftables now)
  • nftables-allports is obsolete, superseded by nftables[type=allports]
  • nftables-multiport is obsolete, superseded by nftables[type=multiport]
• allowed multiple protocols in nftables[type=multiport] action (single set with multiple rules in chain), following configuration in jail would replace 3 separate actions, see https://github.com/fail2ban/fail2ban/pull/2254#issuecomment-534684675
• action.d/badips.py: option loglevel extended with level of summary message, following example configuration logging summary with NOTICE and rest with DEBUG log-levels: action = badips.py[loglevel="debug, notice"]
• samplestestcase.py (testSampleRegexsFactory) extended:
• allow coverage of journal logtype;
• new option fileOptions to set common filter/test options for whole test-file;
• large enhancement: auto-reban, improved invariant check and conditional operations (gh-2588):
• improves invariant check and repair (avoid unhandled exception, consider family on conditional operations, etc), prepared for bulk re-ban in repair case (if bulk-ban becomes implemented);
• automatic reban (repeat banning action) after repair/restore sane environment, if already logged ticket causes new failures (via new action operation actionreban or actionban if still not defined in action);
• introduces banning epoch for actions and tickets (to distinguish or recognize removed set of the tickets);
• invariant check avoids repair by unban/stop (unless parameter actionrepair_on_unban set to true);
• better handling for all conditional operations (distinguish families for certain operations like repair/flush/stop, prepared for other families, e. g. if different handling for subnets expected, etc);
• partially implements gh-980 (more breakdown safe handling);
• closes gh-1680 (better as large-scale banning implementation with on-demand reban by failure, at least unless a bulk-ban gets implemented);
• fail2ban-regex - several enhancements and fixes:
• improved usage output (don't put a long help if an error occurs);
• new option --no-check-all to avoid check of all regex's (first matched only);
• new option -o, --out to set token only provided in output (disables check-all and outputs only expected data).