→ Perplexity is using stealth, undeclared crawlers to evade website no-crawl directives
https://blog.cloudflare.com/perplexity-is-using-stealth-undeclared-crawlers-to-evade-website-no-crawl-directives/
“We observed that #Perplexity [an AI-powered answer engine] uses not only their declared #user_agent, but also a generic browser intended to #impersonate Google Chrome on macOS when their declared crawler was blocked.”
“This activity was observed across tens of thousands of domains and millions of requests per day.”
