I just took a quick look at my server logs. And I see that there are some bots that are constantly crawling my site. If that would be search engines, feed readers etc. that wouldn’t be a problem. But this are crawlers by companies whose websites try to sell me some tools to “make SEO easier” and “uncover the strong and weak points of my sites and their competitors”. They are crawling my site and make me pay for the server while they are building indices to sell for big money? I think that sucks.


@en You could deny those bots access, or serve them a fake procedurally-generated site.

Sign in to participate in the conversation

Fosstodon is an English speaking Mastodon instance that is open to anyone who is interested in technology; particularly free & open source software.