Mirrors/gotosocial
1
0
Fork 0
mirror of https://github.com/superseriousbusiness/gotosocial.git synced 2024-10-31 22:40:01 +00:00
Code Issues Projects Releases Packages Wiki Activity
gotosocial/docs/admin/robots.md
Daenney dcab555a6b
[chore] Update robots.txt (#2856) 
This updates the robots.txt based on the list of the ai.robots.txt
repository. We can look at automating that at some point.

It's worth pointing out that some robots, namely the ones by Bytedance,
are known to ignore robots.txt entirely.
2024-04-22 11:01:37 +02:00

965 B
Raw Blame History

Robots.txt

GoToSocial serves a robots.txt file on the host domain. This file contains rules that attempt to block known AI scrapers, as well as some other indexers. It also includes some rules to ensure things like API endpoints aren't indexed by search engines since there really isn't any point to them.

AI scrapers

The AI scrapers come from a community maintained repository. It's manually kept in sync for the time being. If you know of any missing robots, please send them a PR!

A number of AI scrapers are known to ignore entries in robots.txt even if it explicitly matches their User-Agent. This means the robots.txt file is not a foolproof way of ensuring AI scrapers don't grab your content.

If you want to block these things fully, you'll need to block based on the User-Agent header in a reverse proxy until GoToSocial can filter requests by User-Agent header.