Based on a review of recent failures in my indexing log, roughly 5% of all sites were not being indexed due to indexing errors, and of these indexing errors 62% were sites that had disappeared, 19% were sites where robots.txt blocked indexing, and 5.5% were sites where Cloudflare blocks indexing. Further information at .

Sign in to participate in the conversation

Fosstodon is an English speaking Mastodon instance that is open to anyone who is interested in technology; particularly free & open source software.