bac151cf38
Connection resets are normal, haproxy can and should disconnect from time to time after a timeout. There's no need to log these at the error level, log them at debug level only - but keep logging any other error at that level. Fixes #55. |
||
|---|---|---|
| .cargo | ||
| .forgejo | ||
| data | ||
| LICENSES | ||
| nix | ||
| src | ||
| tests | ||
| .envrc | ||
| .gitattributes | ||
| .gitignore | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG.md | ||
| flake.lock | ||
| flake.nix | ||
| HACKING.md | ||
| justfile | ||
| README.md | ||
| REUSE.toml | ||
iocaine
The deadliest poison known to AI.
Iocaine is a defense mechanism against unwanted scrapers, sitting between upstream resources and the fronting reverse proxy. It is designed to significantly reduce the load caused by the relentless attack of the Crawlers (mostly originating from various AI companies) in a way that does not place undue burden on legit visitors. While iocaine does support presenting a proof-of-work challenge, that should be a last resort, and even if that path is taken, the challenge is inexpensive for the human visitor.
Originally, iocaine started as a garbage generator, a tarpit, a place where the reverse proxy routed unwanted visitors, so that they'd crawl an infinite maze of garbage. Since then, it grew into a more ambitious project, and includes a scripting engine that lets the iocaine-operator tell the software how to treat each incoming request. The idea remained similar, however: to waste crawler resources, while keeping our costs down.
Unlike some similar tools, iocaine does not try to make the bad bots go away, it welcomes them, into an infinite maze of garbage. It does so, because the author observed that if the bots keep being served plausibly legit looking content, they will behave slightly better: they will not try to disguise themselves as much, and they'll keep a steady, but not overwhelming crawling pace. Serving unwanted visitors junk also evoked warm, fuzzy feelings inside the author. He likes warm, fuzzy feelings. The garbage generator used in iocane has been tuned with this in mind, to make it cheap, pretty much on par with serving a static file.
The goal is to have a tool that costs next to nothing for the operator, reduces the load on upstream services, and protects them from the crawlers, while having no (or at worst, very little) effect on legit visitors, but wasting crawler resources at the same time.
For more information about what this is, how it works, and how to deploy it, have a look at the dedicated website. If you're interested in hacking on iocaine, look into HACKING.md to get started.
Lets make AI poisoning the norm. If we all do it, they won't have anything to crawl.