Blocklist mirror

ModeStream only

MetricsUnsupported

MTLSSupported

PrometheusSupported

This Remediation Component exposes CrowdSec's active decisions via provided HTTP(S) endpoints in pre-defined formats. It can be used by network appliances which support consumption of blocklists via HTTP.

Installation from repositories

Setup crowdsec repositories.

Debian/Ubuntu
RHEL/Centos/Fedora

sudo apt install crowdsec-blocklist-mirror

sudo yum install crowdsec-blocklist-mirror

Docker / Podman:

Refer to docker hub

Manual

installation via script

First, download the latest crowdsec-blocklist-mirror release.

tar xzvf crowdsec-blocklist-mirror.tgz
sudo ./install.sh

From source

Run the following commands:

git clone https://github.com/crowdsecurity/cs-blocklist-mirror.git
cd cs-blocklist-mirror/
make release
cd crowdsec-blocklist-mirror-v*/
sudo ./install.sh

Configuration

For manual installations before starting the crowdsec-blocklist-mirror service, please edit the configuration file to add your API URL and key.

The default configuration file is located under : /etc/crowdsec/bouncers/ as file crowdsec-blocklist-mirror.yaml.

If you need to download and restore the configuration file you can find an example on the Repository

Configuration Reference

`crowdsec_config`

Used to nest the configuration related to crowdsec.

Example
crowdsec_config:
  lapi_url: http://127.0.0.1:8080
  lapi_key: ${LAPI_KEY}
  update_frequency: 10s
  supported_decisions_types:
    - ban
  include_scenarios_containing:
    - ssh
    - http
  exclude_scenarios_containing:
    - dns
  only_include_decisions_from:
    - crowdsec
    - cscli
  insecure_skip_verify: false
  listen_uri: 127.0.0.1:41412

`lapi_url`

string

The URL of CrowdSec LAPI. It should be accessible from whichever network the component has access.

`lapi_key`

string

It can be obtained by running the following on the machine CrowdSec LAPI is deployed on.

sudo cscli -oraw bouncers add blocklistMirror # -oraw flag can discarded for human friendly output.

`cert_path`

string

Path to the certificate file used to authenticate with the LAPI.

`key_path`

string

Path to the key file used to authenticate with the LAPI.

`ca_path_file`

string

Path to the CA file used to trust the LAPI certificate.

`update_frequency`

string (That is parseable by time.ParseDuration)

The component will poll the CrowdSec LAPI every update_frequency interval.

`supported_decisions_types`

[ ]string

Limit mirrored blocklists to specific decision types (for example ban, captcha).

Empty/missing list: no type filtering (all types).
Case-insensitive matching.

`include_scenarios_containing`

[ ]string

Ignore IPs banned for triggering scenarios not containing the provided words.

Example
include_scenarios_containing:
  - ssh
  - http

`exclude_scenarios_containing`

[ ]string

Ignore IPs banned for triggering scenarios containing the provided words.

Example
exclude_scenarios_containing:
  - ssh
  - http

`only_include_decisions_from`

[ ]string

Only include IPs banned due to decisions originating from provided sources.

Example
only_include_decisions_from:
  - cscli
  - crowdsec

`insecure_skip_verify`

boolean

Set to true to skip verifying certificate usually used for self-signed certificates.

`listen_uri`

string (<IP>:<PORT>)

The bindable address and port for the component to listen on

Example
listen_uri: "127.0.0.1:41412"

`metrics`

Prometheus metrics

Example
metrics:
  enabled: true
  endpoint: /metrics

`enabled`

boolean

Set to true to enable serving and collecting metrics.

`endpoint`

string

The URI endpoint to serve the metrics on.

`blocklists`

[ ]BlockListConfig

List of blocklists to serve. Each blocklist has the following configuration.

Example
blocklists:
  - format: plain_text
    aggregate: true
    endpoint: /security.txt
    authentication:
      type: none

`format`

string

Format of the blocklist. See the supported values and examples in the Formats section.

`aggregate`

boolean

When enabled, aggregate decisions into CIDR ranges to reduce the size of the served blocklist.

This changes the output to CIDR ranges (including /32 and /128 for single IPs) and makes per-decision metadata unavailable, so runtime filters that rely on such metadata (for example ?origin=) are not compatible.

`endpoint`

string

The URI endpoint to serve the blocklist on.

`authentication`

Configuration used to enforce or bypass authentication on the blocklist.

`type`:

none | basic | ip_based

The type of authentication to impose:

none : No authentication required.
basic : Basic authentication required.
ip_based : IP based authentication required.

`user`

string

Valid username if using basic type.

`password`:

string

Password for the provided user and using basic authentication.

`trusted_ips`:

[ ]string

List of valid IPv4 and IPv6 IPs and ranges which have access to blocklist. It's only applicable when authentication type is ip_based.

`tls`

TLS Configuration is utilized to activate HTTPS on the mirror server.

Example
tls:
  cert_file: /etc/ssl/certs/blocklist-mirror.pem
  key_file: /etc/ssl/private/blocklist-mirror-key.pem

`cert_file`:

string

Path to certificate to use if TLS is to be enabled on the mirror server.

`key_file`:

string

Path to certificate key file.

Global runtime query parameters

?ipv4only - Only return IPv4 addresses

Example usage

http://localhost:41412/security/blocklist?ipv4only

?ipv6only - Only return IPv6 addresses

Example usage

http://localhost:41412/security/blocklist?ipv6only

?nosort - Do not sort IPs

Only use if you do not care about the sorting of the list, can result in better performance.

Example usage

http://localhost:41412/security/blocklist?nosort

?origin= - Only return IPs by origin

Not compatible with aggregate: true (aggregation can’t preserve the original decision origin).

Example usage

http://localhost:41412/security/blocklist?origin=cscli

?supported_decisions_types= - Override decision type filtering for this request

Accepts a comma-separated list or repeated query parameters:

/security/blocklist?supported_decisions_types=ban,captcha
/security/blocklist?supported_decisions_types=ban&supported_decisions_types=captcha

If omitted, the YAML value applies. To include all types, omit the parameter and leave the YAML list empty.

You can then start the service via:

sudo systemctl start crowdsec-blocklist-mirror

If you need to make changes to the configuration file and be sure they will never be modified or reverted by package upgrades, starting from v0.0.2 you can write them in a crowdsec-blocklist-mirror.yaml.local file as described in Overriding values. Package upgrades may have good reasons to modify the configuration, so be careful if you use a .local file.

Formats

The component can expose the blocklist in the following formats. You can configure the format of the blocklist by setting it's format parameter to any of the supported formats described below.

plain_text

Example:

192.168.1.1
192.168.1.2

mikrotik

Generates a MikroTik Script that the device can execute to populate the specified firewall address list.

MikroTik query parameters

Parameter	Description
`listname=foo`	Set the list name to `foo`. By default, `listname` is set to `CrowdSec`.

Example output:

/ip/firewall/address-list/remove [ find where list="foo" ];
:global CrowdSecAddIP;
:set CrowdSecAddIP do={
    :do { /ip/firewall/address-list/add list=foo address=$1 comment="$2" timeout=$3; } on-error={ }
}
$CrowdSecAddIP 1.2.3.4 "ssh-bf" 152h40m24s
$CrowdSecAddIP 4.3.2.1 "postfix-spam" 166h40m25s
$CrowdSecAddIP 2001:470:1:c84::17 "ssh-bf" 165h13m42s

Example: MikroTik import script

Using on device MikroTik scripting following is a starting point to download and import the blocklist. Ensure to adjust the global query parameters according to your needs!

:local name "[crowdsec]"
:local url "http://<IP>:41412/security/blocklist?ipv4only&nosort"
:local fileName "blocklist.rsc"
:log info "$name fetch blocklist from $url"
/tool fetch url="$url" mode=http dst-path=$fileName
:if ([:len [/file find name=$fileName]] > 0) do={
    :log info "$name import;start"
    /import file-name=$fileName
    :log info "$name import:done"
} else={
    :log error "$name failed to fetch the blocklist"
}

F5

Example:

192.168.1.1,32,bl,ssh-slow-bf
192.168.1.2,32,bl,ssh-slow-bf

Juniper

Generates a .txt file with all IP addresses (single host and subnets) in the CIDR notation format supported by the Juniper Networks SRX firewall platform.

Example:

1.2.3.4/32
4.3.2.1/32

SRX Dynamic Address configuration sample

Using the blocklist on a Juniper SRX requires that the published URL ends in .txt. This can be achieved by altering the endpoint config in cfg.yaml as follows:

Sample cfg.yaml:

blocklists:
  - format: juniper
    endpoint: /security/blocklist.txt # Must end with `.txt` for the Juniper formatter.
    authentication:
      type: none

This can then be configured on the SRX firewall as follows:

Sample SRX config:

user@srx> show configuration security dynamic-address | display set

set security dynamic-address feed-server crowdsec url http://192.168.1.2:41412
set security dynamic-address feed-server crowdsec update-interval 30
set security dynamic-address feed-server crowdsec feed-name crowdsec path /security/blocklist.txt
set security dynamic-address address-name crowdsec-blocklist profile feed-name crowdsec

Further information here

A successful configuration should return a similar result when queried:

user@srx> show security dynamic-address summary


Dynamic-address session scan status            : Disable
Hold-interval for dynamic-address session scan : 10 seconds


  Server Name                 : crowdsec
    Hostname/IP               : http://192.168.1.2:41412
    Update interval           : 30
    Hold   interval           : 86400
    TLS Profile Name          : ---
    User        Name          : ---


    Feed Name                             : crowdsec
        Mapped dynamic address name       : crowdsec-blocklist
        URL                               : http://192.168.1.2:41412/security/blocklist.txt
        Feed update interval              : 30       Feed hold interval :86400
        Total update                      : 16310
        Total IPv4 entries                : 16240
        Total IPv6 entries                : 0

Installation from repositories​

Docker / Podman:​

Manual​

installation via script​

From source​

Configuration​

Configuration Reference​

crowdsec_config​

lapi_url​

lapi_key​

cert_path​

key_path​

ca_path_file​

update_frequency​

supported_decisions_types​

include_scenarios_containing​

exclude_scenarios_containing​

only_include_decisions_from​

insecure_skip_verify​

listen_uri​

metrics​

enabled​

endpoint​

blocklists​

format​

aggregate​

endpoint​

authentication​

type:​

user​

password:​

trusted_ips:​

tls​

cert_file:​

key_file:​

Global runtime query parameters​

Formats​

plain_text​

mikrotik​

MikroTik query parameters​

Example: MikroTik import script​

F5​

Juniper​

SRX Dynamic Address configuration sample​

Installation from repositories

Docker / Podman:

Manual

installation via script

From source

Configuration

Configuration Reference

`crowdsec_config`

`lapi_url`

`lapi_key`

`cert_path`

`key_path`

`ca_path_file`

`update_frequency`

`supported_decisions_types`

`include_scenarios_containing`

`exclude_scenarios_containing`

`only_include_decisions_from`

`insecure_skip_verify`

`listen_uri`

`metrics`

`enabled`

`endpoint`

`blocklists`

`format`

`aggregate`

`endpoint`

`authentication`

`type`:

`user`

`password`:

`trusted_ips`:

`tls`

`cert_file`:

`key_file`:

Global runtime query parameters

Formats

plain_text

mikrotik

MikroTik query parameters

Example: MikroTik import script

F5

Juniper

SRX Dynamic Address configuration sample