Here’s a Git repository that gathers information about LLMs and offers some ways to deal with them. The only tool I know how to use is the robot.txt file, which depends on voluntary cooperation. Speaking of, the repository contains an interesting table of bot metrics, which is gathering information about the behavior of these things. For more, follow the link to the repository owner’s blog post on Cory Dransfeldt.