Back

In January 2023, freebsdwiki.net--which has not been actively updated for more than a decade, and whose last update AT ALL occurred in 2021--received 812,706 lines in its httpd-access.log.

In January 2024, that was up to 2,690,389 lines.

As of January 2025, it was up to 7,607,518 lines. And last month--March 2025--it was up to an eye-watering 18,168,053 lines.

Here's a breakdown of user-agent strings seen in March 2025. The top two user agents are Amazonbot and Gptbot. Between the two of them, they account for 35% of all traffic to the site.

That doesn't sound as bad as the chart makes it look... but even the chart doesn't capture the full story.

THIS is what Amazon's scraper traffic looks like: a never ending series of RecentChanges pulls, several times a second, from multiple IP addresses.

This is an INSANELY difficult load to manage, because it isn't really cachable and hits the database HARD.

I hope you didn't expect to see different tactics from OpenAI's gptbot. It's not rotating IP addresses... but I betcha if I block 20.171.207.122 today, tomorrow I see a giant pool of IP addresses behind gptbot just like the one I see now behind amazonbot.

Let's talk about the next few entries on the list:

Semrushbot: this is for an "SEO index", it's been a plague on the net for years.

MJ12bot is for a UK-based engine called "Majestic" which maps links, ignoring actual content. Another very long term, prolific abuser, dwarfed by the current AI scraping.

Petalbot: this one is for "Petal search," which is exclusively used on Huawei phones. Doesn't even have a website!

Barkrowler is another SEO engine focused on links, not content.

After that, we've got several more bots, several demonstrably fake lookalike "chrome" or "safari" agents, and even the first several human-LOOKING user agents are obviously fake... unless you believe, for example, that an elderly version of Microsoft Edge has a greater percent of the traffic share than ALL versions of Safari, Chrome, and Firefox combined.

Bytespider is notable here for pretending it's "mobile Safari" running on Android. Sure Jan.

Anyway, here's another graphical view into the problem: look at what happened in December 2024.

The "unique visitors" stat went up by an order of magnitude between Nov 24 and Dec 24... and not only has it not dropped since, it's gotten actively WORSE.

Not only has the "unique visitors" stat stayed in the six figures level, the *bandwidth* has gone up from typically around 700MiB per month to a whopping TWENTY-SIX GiB per month.

If you're wondering why the stats for February show "zero" visitors and uniques, well, that stats run fired off just prior to the cron job that COMPRESSES all the logfiles--so I'm pretty sure the script failed ENOSPC there, recovering for a little while once the last few humongous logs were gzipped.

Are the scrapers hitting my tech blog also? You'd better believe it--but, lacking the tooling MediaWiki offers to show you just what's changed, they aren't scraping anywhere NEAR so frequently.

After all, they don't want to burn their OWN processing time actually parsing things before they ingest into the latest model, now do they...?

This does raise an interesting question, though... just what the fuck does Amazonbot think it's doing HERE?

@bobthcowboy so, you're wondering why the supervillains aren't monologuing?

I mean, they're NOT GOOD PEOPLE. Good people don't behave like this. There's really no doubt to extend any benefit from, here.

If you really want to listen to supervillain monologues, you can absolutely find plenty of video of the likes of Sam Altman declaring that anything being done in pursuit of AI is worth it, no matter the cost, period.

@jimsalter No, you're right. I'm often involved in hiring in my roles and I actually decided ~10 years ago that I would never hire someone who worked at Facebook at that time or later. And like, there's people right here on Fosstodon who still work for Big Tech, despite the general state of that scene. I don't get it and I try to understand what the motivation is aside from greed or naivety.

@bobthcowboy I won't say I wouldn't hire anybody who worked for Facebook or, say, Microsoft... there was a period in my life when I was actively being recruited by those orgs, and it was DIFFICULT soul-searching figuring out whether the best thing to do was say hell no, or to say yes and try to change the orgs from within.

I would definitely be probing such candidates on morals and ethics during an interview, though.

@spineless_echidna this is one of many, many reasons I am constantly advising people to avoid fail2ban. :)

@jimsalter is there a decent alternative? I'm starting to consider rolling out nepenthes or other tarpits in a parallel VM, but I wonder at what cost.

I guess I could re-write the URL on a per-user-agent basis and just re-direct them to the tarpit. If they're smart enough to detect it and blacklist my genuine addresses, it's still a win.

@spineless_echidna I'm also strongly considering Nepenthes or similar. I'd really, really like to actively POISON those bots with false data, not just hamper them with a tarpitted workload.

You want to train your models? Step right the fuck up, have I ever got some training data for YOU...

@Tubsta @jimsalter @spineless_echidna pissing off the people who know how to do things is always a winning strategy. *Grabs Popcorn*

@oxyhyxo @Tubsta @spineless_echidna might be interesting to run one of those horrible spammer tools that tries to bypass blocks by using a thesaurus on every third word, against some readily-available public domain text that can be ethically sourced.

I'm trying to think of cheap ways to poison the scraped data, too subtly to detect until it makes it into the actual model.

@jimsalter @Tubsta @spineless_echidna thesaurus but 2 or 3 definitions deep. Swap the odd noun for a verb.

@oxyhyxo @Tubsta @spineless_echidna you're definitely picking up what I'm putting down.

@oxyhyxo @Tubsta @spineless_echidna I wonder how hard it would be to just proxy a scraper direct to output from a hosted ChatGPT or similar instance... "incest" in the training data is ROUGH on LLMs.

@matt_garber @spineless_echidna OpenAI and Amazon have both already been caught lying about their user agents and using residential IP proxies (which begs the question: how are they getting ACCESS to pools of residential IP addresses?)when blocked.

I'm looking into deliberately poisoning their models with trash data, personally. You want my CPU cycles and bandwidth? You got 'em, hope you enjoy 'em...

@jimsalter @spineless_echidna Yeah, that’s fair enough, and the poisoned dataset methods are enticing, too. FWIW, from the sample of sites I block the UAs for, some fairly high traffic, I haven’t yet seen a *commensurate* offsetting increase in bogus UAs vs. all the requests I still see coming in from their “genuine” crawler identifier (getting denied), that that’s still my primary method and which doesn’t rely on IPs at all. (I also rate limit everything dynamic to keep reqs/s reasonable.)

@jimsalter @spineless_echidna Re: the performance of doing something with fail2ban or equivalent, the key I learned *if you’re going to* use things like f2b/sshguard is *not* using the default iptables rules configuration where a new rule gets added for each entry, but having a config that uses IP sets (or now, nftables sets) — the performance penalty of adding individual rules goes away, ipsets default to 65k addresses or CIDR range entries and can be increased up, and block lookups are O(1).

@feoh absolutely, if you can manage it. I'm guessing it's following an anchor tag it found on some script kiddie's site, from some attempt said skiddie was making to exploit something along the lines of log4j or similar, but at this point, who the hell knows for sure?

@jimsalter No luck yet but (pardon if this is obvious) did you see this?

There's a specific AmazonBot contact addy in there.

https://developer.amazon.com/amazonbot

@Rairii sure is. This is a mediawiki site, and if you believe AI crawlers don't know EXACTLY what a mediawiki is and hunger SPECIFICALLY for its data... welp.

@jimsalter @Rairii Seeing the same thing on my partner's and my silly MediaWiki site as well that we use for planning stuff. It has barely 50 pages :/

@rubenerd @Rairii the scrapers haven't found another wiki located on the same server yet.

Can't wait until the problem doubles itself. Yay.

@dan @jimsalter of course they do.

@hamoid @jimsalter Oh yes we have. And we still have a lot to do with it. We have a blog post in german about it: https://blog.uberspace.de/2024/08/bad-robots/