Home

The www/anubis port

anubis-1.15.1 – proof-of-work proxy to protect web resources from scrapers (cvsweb github mirror)

Description

Anubis acts as middleware between a reverse proxy and backend web server.
It assesses whether a connection is likely to be from a scraper bot and,
if this seems that there's a chance of this, it issues a SHA-256 proof-
of-work challenge before allowing the connection to proceed.

As of 1.14.x, Anubis decides to present a challenge using this logic:

    User-Agent contains "Mozilla"
    Request path is not in /.well-known, /robots.txt, or /favicon.ico
    Request path is not obviously an RSS feed (ends with .rss, .xml, or .atom)

This should ensure that git clients, RSS readers, and other low-harm
clients can get through without issue, but high-risk clients such as
browsers and AI scraper bots impersonating browsers will get blocked.

When a challenge is passed, a signed JSON Web Token (JWT) is provided
as a cookie, allowing future requests to pass without triggering the
challenge.

Using Anubis will likely result in your website not being indexed by
some search engines. This is considered a feature, not a bug.
WWW: https://anubis.techaro.lol/

Readme

+-------------------------------------------------------------------------------
| Running ${PKGSTEM} on OpenBSD
+-------------------------------------------------------------------------------

While Anubis is intended to be run in a Docker container, it can also be
manually configured and is able to run on OpenBSD. It must be "sandwiched"
between a front-end web server (for example, nginx in reverse proxy mode)
and a back-end.

Example configuration with nginx
================================

Anubis
------
Bind and target ports should be specified in ${SYSCONFDIR}/anubis.env.
Here's a contrived example:

export BIND=127.0.0.1:8923
export DIFFICULTY=3
export METRICS_BIND=:9090
export SERVE_ROBOTS_TXT=0
export TARGET=https://www.openbsd.org:443

Anubis listens for incoming HTTP connections on 127.0.0.1:8923, uses a
medium difficulty proof-of-work, and forwards connections to a backend
running at https://www.openbsd.org:443. It provides Prometheus-style
metrics at http://:9090/metrics (not bound to a specific
address here, but that is allowed).

Typically TARGET would be pointed at a backend server which is either
hosted locally, or hosted remotely but firewalled off so that bot traffic
can't reach it directly.

Front-end nginx
---------------
server {
	listen		443 ssl;
	listen		[::]:443 ssl;
	http2		on;
	server_name	somehost.example.org;

	ssl_certificate		/etc/ssl/server.crt;
	ssl_certificate_key	/etc/ssl/private/server.key;
	ssl_protocols		TLSv1.3 TLSv1.2;
	ssl_ciphers		'TLSv1.3:TLSv1.2+AEAD+ECDHE';

	location / {
		proxy_pass http://127.0.0.1:8923;
		proxy_set_header X-Real-IP $remote_addr;
		proxy_set_header Host www.openbsd.org;
		proxy_http_version 1.1;
	}
}

Maintainer

The OpenBSD ports mailing-list

Only for arches

aarch64 amd64 arm i386 riscv64

Categories

lang/go www

Build dependencies

Files

Search