Mails were not going through, and the whole server was throwing strange errors. After some investigation I ran df -h on the server, and suddenly the problem was clear.
What the numbers showed
Over 24 hours, git.opensocial.at, a public Forgejo instance mirroring the Friendica repositories, served:
| Requests | 745,104 |
| Egress | 175 GB |
| Forgejo container CPU, 24 h average | 127 %, i.e. 1.3 cores, sustained |
context canceled log lines per day | up to 1,730,000, trending up |
Later peaks reached 2.2 to 3.6 cores, continuously, on a box that also runs
about a hundred other containers.
Two incidents came out of this. The disk above had been filled by a runaway repo-archive directory that reached 573 GB, which is what took mail down. The log flood was the second one, running straight at a 30 GiB retention quota.
What made the logs explode
The log flood had little to do with request volume. There were only about
130 /compare/ requests per day.
Two things multiplied them into millions of lines.
A pathological endpoint. A compare page across the full history is 11 MB and
takes about ten seconds to render. (The upstream served the same page at 136 MB in 49 seconds.)
A logging bug. When the client gives up and nginx records a 499, Forgejo logs one error line per commit iteration instead of once per request:GetLatestCommitStatus, GetUserByEmail for signature checks, GetTagNamesByRepoID. A single aborted request produced roughly 26,000 lines.
Measured over six hours: 505,308 error lines.
Of those, 505,200 came from 19 aborted requests. 108 were real errors.
Who was doing it
Two distinct populations, and they need different answers.
One crawler announced itself. meta-externalagent accounted for
239,570 requests, 32 % of all traffic, about 37 GB.
It uses its real name and it ignores the robots.txt Forgejo ships by default. Its top paths were precisely the ones that file disallows:
/*/*/blame/ — Disallow
/*/*/commits/ — Disallow
/*/*/src/ — Disallow
/*/*/raw/ — Disallow
The rest was a residential proxy botnet. Every user agent used two or three
times. Uniformly outdated Chrome versions: 67, 75, 80, 86, 89. Every request from a different home IP in a different country.
Why the obvious defences missed
Each of these was a reasonable first idea. Each failed for a structural reason
rather than a tuning one:
| Defence | Why it does not work here |
|---|---|
| Per-IP rate limiting | Each IP appears once or twice. The limit never triggers. |
| User-agent blocking | Catches the one honest crawler. The botnet spoofs real browsers. |
robots.txt | Ignored by both populations. |
| fail2ban, IP bans | Rotating residential IPs, so the ban list never catches up. |
| Disabling source archive downloads | Hides the UI button only. The /archive/ endpoint has no such check, which I confirmed in the source. |
Why this instance and not the upstream
The upstream, git.friendi.ca, had been running Anubis for a while:
$ curl -A "Mozilla/5.0 ... Chrome/145" https://git.friendi.ca/
set-cookie: techaro.lol-anubis-auth=…
<title>Making sure you're not a bot!
The mirror had nothing. The crawlers were not targeting it. They were crawling broadly, bouncing off the original, and finding the identical content here, unprotected.
So for me, in the future, before mirroring popular repos I will look into how the upstream is protected, and set hard limits (like I did for the whole Forgejo instance) so it cannot take the whole system down.
Putting Anubis in front
Anubis issues a proof-of-work challenge to browser clients. A person solves it once and never notices it happened.
The mechanism is economic, not absolute. Proof-of-work does not make mass crawling impossible, it makes it expensive: the cost lands on the client instead of on the server, and it scales with the number of pages fetched. A distributed crawler can still pay that cost if it decides to. What changed here is that it stopped being free, and everything else on this page depended on identifying the client first.
Deployed 2026-07-21, v1.25.0, since bumped to v1.26.2, inline in the stack, in front of the nginx proxy:
Traefik ? Anubis ? nginx ? Forgejo
Some decisions that mattered more than the deployment itself:
Inline per service, not a shared Traefik middleware. Anubis’ cookie domain is singular, and this host sits under a different apex domain than the rest of the platform, so a shared middleware would need one instance per apex domain anyway.
One instance per protected service is also the upstream recommendation.
An explicit allowlist above everything else. Git clients, the container registry, Git LFS, the write API and the RSS feeds are never challenged. Getting this wrong does not degrade the service, it breaks it: a challenged git clone is a
failed clone. Git over SSH bypasses Anubis entirely.
bots:
# git clone/fetch/push over HTTP (UA git/*, libgit, go-git, JGit)
- import: (data)/clients/git.yaml
# Container registry: docker pull/push, Renovate, CI. Path-scoped, any UA.
- name: allow-container-registry
action: ALLOW
path_regex: ^/v2(/|$)
# Git-LFS: not covered by git.yaml
- name: allow-git-lfs
action: ALLOW
path_regex: ^/[^/]+/[^/]+(\.git)?/info/lfs/
# Write API (non-GET/HEAD under /api/): tokens, CI
- import: (data)/common/allow-api-like.yaml
# RSS/Atom feeds
- import: (data)/apps/gitea-rss-feeds.yaml
Order matters here. The first rule that matches decides, so anything that must never be challenged has to sit above everything else in the file.
Three populations, three answers. Crawlers that harvest in bulk for search indexes or training are denied outright. Clients that spoof a browser get the proof-of-work challenge. Well-behaved search engine crawlers are let through.
That last distinction took a correction. Anubis ships a convenient meta rule set that blocks, in its own words, “all AI/LLM associated user agents, regardless of purpose or human agency”. Importing it looks like the obvious choice, and I had it in for weeks. It also bundles a list that denies ChatGPT-User, Claude-User and MistralAI-User, which are not crawlers at all: they are fetches a person triggered by asking an assistant to look something up. Blocking those turns away readers, not scrapers. The three crawler lists are now imported directly and that client list is left out.
One client stayed blocked after the change, and it is the more interesting case. Perplexity-User is not in the list I dropped. It matches the general AI catch-all, whose expression covers the vendor name rather than a named agent, and the upstream comment says why: Perplexity documents its crawlers but does not say which of them collects training data, so the whole name is treated as one. That is a defensible call by the people who maintain the list. It is also a call I did not know I had made until I sent the request and read the response. An imported rule set decides things its name does not announce, and the only way to find those edges is to try the clients you actually care about.
# AI crawlers. Imported individually, NOT via (data)/meta/ai-block-aggressive.yaml:
# that meta file also pulls (data)/clients/ai.yaml, which DENYs ChatGPT-User, Claude-User, MistralAI-User, Perplexity-User — human-triggered fetches, not crawlers. Blocking those blocks readers.
# Perplexity-User stays denied regardless: ai-catchall matches the vendor name, not a named agent.
- import: (data)/bots/ai-catchall.yaml
- import: (data)/crawlers/ai-search.yaml
- import: (data)/crawlers/ai-training.yaml
The third answer, the challenge, comes from no rule that names the botnet. Nothing names it. Anything claiming to be a browser gains weight, and the weight decides:
- name: generic-browser
user_agent_regex: >-
Mozilla|Opera
action: WEIGH
weight:
adjust: 10
thresholds:
- name: minimal-suspicion # weight 0: git, docker, curl, CI
expression: weight <= 0
action: ALLOW
- name: moderate-suspicion
expression:
all:
- weight >= 10
- weight < 20
action: CHALLENGE
challenge:
algorithm: fast
difficulty: 2 # two leading zeros, very fast for most clients
A real browser and a spoofed one both land at 10 and both get the challenge. The real one solves it in the time it takes the page to appear.
What changed
| Before | After | |
|---|---|---|
| Forgejo container CPU | 2.2 – 3.6 cores | 0.02 cores |
That is about 99 % less, immediately after the deploy, measured with cAdvisor:rate(container_cpu_usage_seconds_total[5m]).
Anubis issues around 1,391 challenges per minute against the ongoing crawl, so the traffic itself has not gone anywhere. It is being absorbed at the proxy, and the application no longer sees it.
Things that bit
A policy import that only exists on main. The bundled policy referenced a crawler definition added after v1.25.0. In the pinned release image the file is absent, the config fails to parse, and Anubis crash-loops. Removing the import fixed it, and the file does exist in the version running now. The rule I took away: check policy imports against the data tree of the image tag you pinned, not against main, or your own defence changes behaviour without asking.
The challenge returns HTTP 200 by default. That looks wrong and is deliberate.
Aggressive scrapers want a 200 and are more likely to move on when they get one. Left as-is.
The fix reproduced the original problem. At INFO, Anubis logs one line per challenge, roughly two million lines a day, which is the same log-flood class the whole exercise started with. Set to WARN. Warnings and errors still surface, and effectiveness is tracked through the CPU metric.
Fixed the incident, introduced the same issue again.
What this does not solve
Every browser visitor is challenged once, including humans, so cookie and session behaviour needs checking, particularly on iOS where tracking prevention interferes.
And this defends one service. Anything else publicly reachable and expensive to render is still standing there with the door open.
The whole process was a big lesson about AI crawling. I have since talked to other service providers about how to mitigate it, and a lot of them have already installed systems like Anubis. For me, since I use AI both at work and privately, it was important to slow the crawlers down, not to forbid them completely.
