Over the last 72 hours, Claude-searchbot [1] alone fetched ~205,000 pages. Sent exactly 1 referral. There is a lot of free financial data on the site, hoping for real users to benefit from it. It is hard to not feel a little cheated out that Claude gets to claim "Found it!" to its users without me getting no credits or compensation whatsoever.
[1] Exact user agent `Claude-SearchBot/1.0; +searchbot@anthropic.com)`
Proof: https://i.postimg.cc/Pqc3SS8T/Screenshot-2026-08-07-at-5-33-...
Of course a sophisticated scraper can "figure out" how to bypass. It isn't trying to be foolproof, it's adding an extra cost to deter massive amounts of bot traffic.
I put it in front of my hobby project because I can't afford to serve hundreds of thousands of bot requests from residential proxies all across the world, and I didn't want to route all of my traffic through a third party company like Cloudflare.
I've been happy with Anubis.
2. I'm a real user and I have to enable JavaScript and run arbitrary programs from untrusted websites to access them. One browser that I use doesn't even support JavaScript (w3m). Fortunately it is unaffected because it doesn't have the string "Mozilla" in its UA and Anubis lets every user without that pass to not hurt legitimate bots like search engine crawlers. As said Anubis is trivially bypassed. The only reason it isn't bypassed is because it's simply too irrelevant to the AI companies.
For a personal site where you don't care about 0.1% of your users getting blocked, it's probably fine. For a business, blocking some of your paying customers isn't acceptable.
That depends on your business and the costs you're incurring from the bots. It might be justifiable to lose 0.1% of your customers if it means saving server costs from serving useless bot traffic.
The fact that this trade-off isn't obvious indicates you haven't tried to run a site at scale and make a living off of it.
Which one then wonders why they would install it in the first place.
that is a no-no square in judeo christian societies
Perhaps you should check if anybody else sees what you see. After browsing through every image I could find in the history of this project, I, at least, am convinced: any "sexualization" _you_ see is entirely in your head.
> ... in judeo christian societies
I really hope you're not trying to whitewash your own bias by alluding to those religions. From everything I've read of their texts, and interacted with practioners of those religions, I'm unaware of any bias that classifies _cartoon depictions of children_ as "sexual".
Perhaps you should try whitelabeling instead. Don't like the packaging? Package it yourself!
Most people either don’t know or don’t care about “those subcultures”. I bet a lot of older people think it’s a cartoon figure of Betty Boop (nurse) and miss the furry bit since it appears and disappears quickly. Most people also don’t have a brand.
So it seems like you’re describing a concern that only affects a tiny fraction of people:
- Not interested in paying for custom branding, so obviously not a corporation or influencer
- Dislikes cartoons
- Aware of, and hostile towards, “furry” subculture
That has to be an exceedingly small fraction of potential users of Anubis, and given how much businesses and branders will pay to custom-brand something, I’d counsel them to stay the course. Sure, a few never-payers will never pay, but they wouldn’t have anyways, so they can cope with Nurse Betty or look elsewhere for a competing free product.
If you think about this in physical market square terms — in other words, a bazaar — it seems horrendously rude to complain about a shop logo sticker on a free product handed out to anyone that walks up and asks for it. If you want it white-labeled so you can write your own name/logo on it, you pay for the privilege of displacing their name with yours. But you don’t stand there and loudly complain that their shop mascot has dog ears while holding a freebie bag of product, without losing the respect of everyone who hears you doing so.
Also this view seems a bit elderly. For most towards the end of the millennial curve and younger, anime is no longer subculture, its just general culture at this point. Although I would agree it isnt necessarily what you'd want for every platform and web presence.
If you can't be bothered to maintain a trivial fork, then why should the author of anubis be bothered to serve your branding needs? It's not like you have a service contract or anything do you?
So, one can put in the minimal effort of removing one div.
> being associated
There's no requirement of association of any kind. See above and below.
> request
Cool. Pay for your own branding. Request fulfilled. No? Then it's a demand, not a request.
Try opening gcc bug tracker on your phone: https://gcc.gnu.org/bugzilla/
You're complaining about the symptom (needing to have these checks) not the cause (if they don't, 99% of their traffic will be bots, the site will slow to a crawl and be unusable anyway).
In any case, I saw the dumb anime girl for about 2s then the site loaded. Not a big deal.
It's still valuable to raise the cost of scraping of course. I don't think anything can really stop a determined scraper from impersonating a human. I wonder though if a system similar to Anubis but mining some crypto would make bots _welcome_ - since they're paying for their traffic.
Then they'd be blocking themselves from the website.
To help you combat these unwelcome scripts, we’ve partnered with Disconnect and compiled lists of domains that serve cryptomining scripts so you can block them directly through Firefox. When you download Firefox, tracking protection is enabled by default, blocking known cryptominers. pow_server http://127.0.0.1:8080 backend avx512-x16
──────────────────────────────────────────────────────────
uptime 00:03:12
solver ● BUSY difficulty 9, 0.3s
queue [####################............] 5/8 peak 12
──────────────────────────────────────────────────────────
accepted 1240 solved 1180
503 shed 48 504 timeout 2 4xx/5xx 10
──────────────────────────────────────────────────────────
last difficulty 5 nonce 645376 in 9 ms (101.6MH/s, avx512-x16)
hashes 3.90GH total avg 65.3MH/s
Ctrl-C to stop
While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.
The main author is aware that the algorithm is far from ideal for this purpose. See https://news.ycombinator.com/item?id=48869064 . If more bots start to answer the primitive challenge anubis uses now, that'll hasten implementation of a different algorithm.
Don't let the perfect be the enemy of the good enough. For now, the algorithm or challenge scheme almost doesn't matter. Since it's much smaller-scale than cloudflare's challenges, that's probably why very few scrapers and botnets bother to solve anubis's trivial sha2 pow.
Targeted attacks may not be repelled at all. That's not the point.
Checks for existence of a specific static cookie, if it does not exist, output a small page that sets the cookie via JS and reloads. Sadly this kills Noscript, but it would be possible to add a form in <noscript> that when submitted sets the cookie serverside.
Is this trivial to bypass? Yes. It still keeps out 95% of unwanted bots. Reality is most do not target you specifically they just want to mass-scrape with low effort. Running headless browsers is way more expensive for their op
I've extended this with a FCRDNS checked exclusion for Googlebot.
Another quite effective measure I figured out was checking the existence of Sec-Fetch-Dest header if the User-Agent claims to be a modern browser. If you don't want to close down too much.
Also, I only apply these rules to routes that are not cheap and cached.
For a webapp that sets a long-lived cookie, that cookie could be used to bypass anubis completely, or lower the weight in anubis so that it doesn't send its pow challenge unless there are major red flags. If bots start to abuse that exception, it can be removed.
They also don't load CSS but for some reason the security theater PoW won the mindshare.
Define "scale". For any reasonable wait that you're willing to impose on your users, any PoW scheme heavily favors attackers. They have unlimited time and can be scraping even while they're asleep. Your visitors on the other hand don't have that luxury. You might argue that's not the point and it's only to stop dumb scrapers that are effectively ddosing your site, but if it's just dumb scrapers, you could've stopped them less onerous measures like tls or javascript fingerprinting.
sha256(challenge + ascii(nonce)) means that the first compression round of the function can be cached and the second compression round is just the nonce plus the cache. This is the same trick used in Bitcoin mining and would have been avoidable by putting the nonce first, so immediately any non-naive code has to do half the proof of work as the vanilla solver.
write a implementation of the anubis proof of work in native c code, optimized for speed above all else. use every trick available to make the proof of work as efficient and fast as possible, including modern processor tricks on the x86 platform. your code should avoid using external libraries where possible, include tests, and be readable and concise. a reference for what needs to be met is in this repository. https://github.com/TecharoHQ/anubis
let’s develop this more. turn this solver into a local HTTP server that can be given work in the request, and it returns solved work. make an end to end tester that sends test work to the solver and waits for a valid response. add support for solving with a GPU using cuda.
In addition, nothing prevents anubis from sending a wasm solver instead of js, reducing the gap between a custom native solver and a js solver.
The big blocker is that testing against dozens of googles chrome to ensure functionality on abandoned smart TV oses takes a nontrivial amount of time. As an example of the level of debugging and the like required: https://github.com/TecharoHQ/anubis/pull/1684/changes/67621f...
My office gets very warm when chromesweep runs.
This is something that is complicated enough that even though LLM tools can help, it's not a magic bullet. It's just complicated in general.
This is D1 - which has very surprising costs. you may just want to drop D1 and move to a static site. There’s no reason your site should cost this much.
My allegedly more complicated VPS stack needs an afternoon of attention every two years when a new Debian major release is necessary, and costs have been predictable for 15 years, no matter what happened traffic wise.
Loot first, ask questions later.
It makes sense that some crawlers, in the style of Google, would want to index the entire internet. But what is the point of the same crawler re-fetching a page they already fetched an hour ago? Or possibly all this traffic is just independent entities, each trying to cache the internet? The scale of bot traffic makes this seem unlikely.
What's the motivation behind the same entity re-fetching a page it just fetched less than an hour ago?
Nuance, complexity, and an awareness of 'other people' do not exist in their worlds.
They all shifted from mostly paying for themselves (or being so cheap it didn't matter), to essentially producing zero income while resource usage leapt up in magnitudes
I couldn't justify the stress and hassle of making sites, that a few dozen people a day might find useful, into some complex hyperscalable obligations just to feed the bots
Type something stupid, it "visits" 200 websites and regurgitates some random gibberish.
It's far from perfect, but it was a quick way to get rid of bots, while not completely blocking people from countries such as Vietnam.
However, on a Gitlab instance I manage (500 users), we have to restrict viewing of git logs and pretty much everything except issues. The bots were too aggressive. Chinese crawlers have access to a huge range of IPs and they often do only 10-20 requests per day, while generating in total over 50k requests per day. Our server load went from 99% down to 0.1% after that (and it's a fairly big server).
Kinda sounds like we're missing a peer to peer network here.
Instead of downloading the same data over and over again we can just download it once and then share it.
Wouldn't that be better for everyone involved?
It would also function as a distributed WayBack Machine, in case anything ever happens to the Internet Archive. (Which I think is desperately needed, bot apocalypse aside.)
The domain name system is now just a rent seeking scheme. It was great as a temp solution but over time it has deleted more content than preserved. I might in theory be billed for having a country name but I don't pay for a city, street name, house number or postal code. Online you should be able to move your widget shop to widget street. Can bill people who want to live on real estate street or used car street and/or set some requirements.
Is it really a bot if it's in response to a human asking for some aggregate information about charities, triggering a web search and then following the result links to get details for the human? Well, clearly yes it is, but it's a very different proposition from this article's implication that "they have no throttling on their scrapers"[1].
> Challenge 46 datacenter ASNs. Humans don't browse from AWS.
People who have workstations in the cloud do.
> The bots use 99% of the bill and I pay 100% of it.
Running a site this way is always a wallet-DDoS risk.
[1] though yes, by far most will be pure automation with no human in the loop. It's an assumption on my part, but feels like a safe one.
What do you mean "this way". What other way is there to run the site?
That's not absent from the article, it's right there in the section titled "The Claude ratio". ChatGPT, Claude, etc. use different user-agents for scraping vs user-initiated requests, and the author notes that user-initiated requests were an absolutely miniscule fraction of the total traffic.
Yeah, googling it does seem like "Claude-User" is for user-initiated requests.
By "Claude sent me" I thought the author meant referer header in real browser requests showed that they came from.
I mean, that's what the section "Pages crawled per visitor referred" refers to, right?
2 access.log [1]
40 botpoop.log [2]
Unfortunately, the best practice is to make css cacheable so you need to keep long histories.
“For a low $/GB, we’ll give you everything from this site and 1000 others as (structured data/a database)!”
(yes there are lots of good counter arguments to this, but before you reply think ahead a couple extra steps)
Are you speaking from experience, or inferring from articles like this?
I serve a static site on the lowest Linode $5/month VPS and it is grotesquely overprovisioned for that use case. It is not the case that every site is getting slammed every second by hundreds of requests per second.
Now, if you have some sort of dynamically-computed website that is generated by a slow scripting language that is poorly optimized and hits the database too many times for a single page, yeah, it doesn't take many RPS to take you out. But that isn't the only option; it's the slowest of the slow options. Realistic, there are plenty of sites that match that description, but I concatenated that many clauses on purpose. Drop any one of them and your personal site will be fine.
I survived handling the search search traffic generated by this thread[1] on PC hardware off a residential broadband connection without any sort of degradation. Only time I've gone offline from traffic was when Elon Musk tweeted a link to one of my blog posts, and that was just a short temporary blip.
I had a client ask me to look into why their bill spiked 5x (admittedly not by much - from a few dollars to tens). They run a static site, and it turned out someone had inadvertently replaced an image with a massive uncompressed photo. Every unique visitor was caching O(25MB) when they loaded the homepage. Also funny because now everyone has 100Mb+ internet, nobody noticed a few extra seconds on load.
There are lots of solutions to this if you actually need to serve a lot of bytes, but you can get stung even with very simple deployments. Not everyone wants to put Anubis or Cloudflare in front of their site.
The next tier up from free is $25/month or $240 per year.
https://www.cloudflare.com/plans/ https://www.cloudflare.com/plans/free/
But if you want them to actually stop, you can also just serve a little JavaScript page that sets a cookie and refreshes, to anyone who hasn't set the cookie. The DDOS attacker doesn't run JavaScript.
I made this tool to try and make it easier for beginners to get something up on the public web for free: https://weejur.com
The vBulletin and PHPbb style forums have issues with slowdown (I haven’t had a forum since 2015; even back then those forums were overrun with spambots), but static content on a nginx site can be served lightning fast.
I have frequently have had to update a GitHub page, push the change, and then GitHub’s actions puke instead of deploying the change. The workaround is that I have a .txt file with a list of GitHub actions which failed, and when GitHub actions fails, I update that .txt file and push the updated site, which GitHub actions will hopefully successfully deploy.
GitHub pages are OK for pages which aren’t updated very frequently, but they are not OK for pages which update frequently.
If you can't, well then... We're stuck.
Because you have no CDN, all the bots are coming directly to you.
Because that's no CDN or distributed WAF, they're hammering your box.
Because your IP is directly exposed, you're also showing up on Shodan.
Your server may have enough CPU and RAM to handle the load, but does your monthly bandwidth allocation with your provider?
Many are finding previous bandwidth limits (starting around 10TB/month) are no longer enough.
My single static webpage with no updates in 3 years is doing that, which is (one of the reasons) how I end up where that site (and many others in business and personally) is.
You're chasing a dream for a world that doesn't exist anymore.
This is a small booking app without any useful information at all, it surprises me that the AI boots have no discernment about what the are scraping, just wasting their own and other peoples resources. And their own reputation! You would thing they could spare a few tokens on a classifier model to do a quick evaluation of their scraping efforts, but apparently they do not.
Anyway, I have done my best to block these UAs and so far it seems to have improved the situation.
https://developers.cloudflare.com/bots/additional-configurat...
my take with all the bots - the web is gonna be a bunch of private walled gardens. with most sites set to no index. you will only discover them via referral from someone real.
The problem, to recap the title, is that "99% of My Website Traffic Is Bots".
How to fight all of those bots, all of those web-scrapers...
If the sole underlying issue is bandwidth for one's website (someone doesn't have enough of it and/or they pay too much for it), then one possible solution to this might be for a single tech company to create an online cache of web pages, accessible to all bots as an alternate route to fetch the desired web pages.
If a tech company (or consortium thereof) decided to do this, they could certainly charge money to each AI / bot utilizing the service...
Which means that it might be an investable idea as a for-profit service...
Now, if the underlying issue is the privacy of individual web pages (or relative privacy, as the case may be!) from the public internet, then perhaps the solution is to simply put those behind user logins and/or a gauntlet of tests (which could also be tied to login!) designed to exclude AI/bot web traffic from human web traffic.
Consider what some BBS'es and the online services of yesteryear used to do (Compuserve, The Source, Prodigy, QuantumLink, AOL, etc.)... basically they'd get a user's mailing address, and then physically snail mail them their login and password to their physical home address...
Oh sure, user registration wasn't as fast back then as it is today... it might take several days to receive your username and password via postal mail -- but as a system operator you were 99.999999999% guaranteed that when you saw such a login on your BBS, that it was an actual live human user...
There may be a market for a service like that, too...
That is, validate that actual users are actual users, and give them some kind of credential that can be checked by an actual web site... find a way to do this at scale...
Anyway, just thinking aloud... a very interesting article, and a very interesting problem to have!
I stopped posting to my website. Why should it be so much work to stop this theft?
Would it be helpful to have geofencing and regulation?
Are bots the solution to nosy websites that want to know things about their visitors?
Not clear how the author arrived at the precise 99% figure; perhaps "99%" is a figure of speech
"And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers."
"I'm trying to run a business here."
What's the business
(where "business" is defined as "buying and selling")
From https://patronview.com/robots.txt
# As a condition of accessing this website, you agree to abide by the following
# content signals:
# (a) If a Content-Signal = yes, you may collect content for the corresponding
# use.
# (b) If a Content-Signal = no, you may not collect content for the
# corresponding use.
# (c) If the website operator does not include a Content-Signal for a
# corresponding use, the website operator neither grants nor restricts
# permission via Content-Signal with respect to the corresponding use.
# The content signals and their meanings are:
# search: building a search index and providing search results (e.g., returning
# hyperlinks and short excerpts from your website's contents). Search does not
# include providing AI-generated search summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
# augmented generation, grounding, or other real-time taking of content for
# generative AI search answers).
# ai-train: training or fine-tuning AI models.
# use: how AI systems may consume the content (immediate, reference, or full).
# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
If, for example, the website operator had enforceable intellectual property rights in the "content", such as copyrights, then perhaps the operator could restrict access to the "content" under the threat of litigation to enforce those rights
Basic questions
1. Is the "content" protected by intellectual property rights, e.g., copyrights
2. Does the website operator have intellectual property rights in the "content", e.g., copyrights
3. Does the website operator have agreements with the rights holders, e.g., granting the operator authorization to restrict access to the "content"
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /secrets.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /credentials.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /secrets.yml HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /service-account.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /key.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:31 -0700] "GET /config/.env HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /service_account.json HTTP/2.0" 404 366 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /serviceAccountKey.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /firebase-adminsdk.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /Dockerfile HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.github/.env HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /firebase-service-account.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.docker/config.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.npmrc HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.boto HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.s3cfg HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.svn/entries HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.htpasswd HTTP/2.0" 404 346 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /terraform.tfstate HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /docker-compose.yaml HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.vscode/launch.json HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_rsa HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_ed25519 HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/id_ecdsa HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/authorized_keys HTTP/2.0" 404 343 "-" "anthropic-ai"
34.124.XXX.XXX - - [07/Aug/2026:07:55:32 -0700] "GET /.ssh/known_hosts HTTP/2.0" 404 343 "-" "anthropic-ai"Chatbots have super charged this dynamic though, to the point that it is showing up in the robots.txt data. The last few weeks I've been having Claude rerun some old analysis of Common Crawl from back then, when I have spare usage and time. What I've found is that you can see pretty clearly the rise in people outright blocking AI chatbot related crawlers likely because of how aggressive they have become.
Quarter Crawl GPT Claude CC G-Ext Byte Bing Google
2023 Q1 2023-06 0.00% 0.00% 0.16% 0.00% 0.06% 0.47% 0.39%
2023 Q2 2023-14 0.00% 0.00% 0.18% 0.00% 0.06% 0.45% 0.38%
2023 Q3 none — — — — — — —
2023 Q4 2023-40 2.21% 0.00% 2.12% 0.04% 0.11% 0.39% 0.27%
2024 Q1 2024-10 0.53% 0.05% 0.31% 0.09% 0.18% 0.34% 0.31%
2024 Q2 2024-18 0.55% 0.09% 0.32% 0.11% 0.24% 0.32% 0.31%
2024 Q3 2024-30 0.68% 0.22% 0.36% 0.20% 0.38% 0.24% 0.33%
2024 Q4 2024-42 1.10% 0.50% 0.44% 0.32% 0.50% 0.25% 0.40%
2025 Q1 2025-05 1.14% 0.66% 0.54% 0.42% 0.66% 0.25% 0.44%
2025 Q2 2025-18 1.37% 0.93% 0.63% 0.70% 0.92% 0.29% 0.19%
2025 Q3 2025-30 1.42% 1.07% 0.74% 0.62% 1.01% 0.31% 0.27%
2025 Q4 2025-43 1.92% 1.51% 1.23% 1.15% 1.52% 0.27% 0.19%
2026 Q1 2026-04 2.13% 1.76% 1.68% 1.58% 1.77% 0.22% 0.15%
2026 Q2 2026-17 2.80% 2.38% 2.26% 2.13% 2.50% 0.22% 0.14%
2026 Q3 2026-30 3.45% 3.01% 2.89% 2.71% 3.16% 0.21% 0.14%
It's very clear that website owners do not like getting their content scraped and are indicating to GPTBot et al. that they are not welcome. It's a shame that CCBot is caught in the cross fire, but that's life. Bing and Google are doing just fine though, almost like having significant power in the search engine market gives you an advantage in other markets too. Who knew!
> And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds.
Being self aware does not make it okey. Either you are okey with scraping (like me) or against it. Don't use it yourself and block your site at the same time.
These same people will be crying about how Cloudflare ruins the internet because they get these captchas.
There it is. Just like the fckn spammers who ruined SMTP email, scrapers externalize the costs.
Who finances the effort to use residential proxies? That takes a lot of effort, even if it's shoddy
> And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch.
> But those 4,000 bots were just the warm-up.
I just hate this style of writing like you're on Twitter. Why does the above need to be 3 different paragraphs? A paragraph break indicates a separate thought but the author is still talking about the same data and still making their point. The sentence "But those 4,000 bots were just the warm-up." is effective when still the last line of a paragraph and it signals respect for your readers. I stopped reading after this because it's just a terrible reading experience.
Here's a correct version that doesn't read like the author left for a week to think about what the next sentence would be or having some sort of anxiety-induced mental pause:
> Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer, which is usually the easiest way I spot a bot. They were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. But those 4,000 bots were just the warm-up.
https://patronview.com/news/99-percent-of-my-website-traffic...
And you're totally right: I mostly post on X (nee Twitter) and I probably have ADHD or just a low attention span, so I prefer to read things broken up into paragraphs. But for a smarter audience like this, and that reads long-form blog posts, I should tighten it up.
Thank you for the suggestion. LMK any other edits and I'll be happy to tighten it up.
For other edits I think my qualm applies to other parts of the article but I found that specific paragraph the most impactful way to illustrate what I was talking about. I leave the rest to you as a challenge. Don’t lose sleep over it, there will be more writing in the future to apply it to.
As for the ADHD stuff and the urge to self diagnose consider something less severe but similar symptoms. Have you considered VAST? Here is a good HN comment briefly detailing it and mentioning a book (titled ADHD 2.0 I believe) that may be more in line. I am not a doctor of course and VAST is rather new.
https://news.ycombinator.com/item?id=49035436
Anyways I will finish reading your article now since you’re so wonderful to take a bit of feedback and be proactive.
(This isn't a condemnation. AI can often do a better job of representing thoughts than humans.)
Examples:
- The article organization
- The general language flow
- The bullet points with a bolded gist, a colon, and then elaboration (and bonus with details stats).
- The images are almost certainly AI generated. They look AI generated.
Using it slowly sucks the uniqueness out of you.
> A human on a VPN sees one CAPTCHA and passes, but a headless browser fleet sees a wall.
I do wish we’d stop complaining about em dashes though, that’s lazy criticism. The sentence structure is far worse, and a big tell is subheadings that are all variants of “The <adjective> <noun phrase>”.
"Use only ASD-STE100 Simplified Technical English when communicating with me or writing documents, comments, and other communications intended for humans."
As a result, I've never seen Claude or Codex use it's weird terms like "load-bearing".
But yes your other points are absolutely right
(i skimmed the whole post, there are none)
Or maybe -- just maybe -- using dashes isn't a sign of content being AI written, just a style it picked up from the training data.
Never fails.
The OCD part of me can now go about my day.
A second side effect of a knee-jerk reaction to bots crawling websites is that if you try to fight all bots, you also end up hurting real users that use "bots". If I run a local script or an LLM that needs to fetch a web page from your site and you block that script or LLM as being a bot, you hurt me, the user. I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website. That might or might not be what you expected, but it's worth taking into account.
And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway. Bot traffic is not necessarily bad.
No; in this case you are not a user, you are a bot user.
For instance, I wanted to buy a piece of equipment the other day from a local company for a specific usecase. I wanted to find a specific price/weight/specs ratio, and asked an llm to loop through the 20 or so items, fetch their page and calculate and present some values for each. This then led to me going and buying the one I found. So the llm was mainly just an extension of me clicking into every page and making a spreadsheet myself.
However, if it were to continuously poll, or just scrape or something with no intention of buying, I would be no better than a bot.
This is giving "why is this plane full? What in doing is important but what is everyone else doing here?"
To be fair, I do not generally support wholesale banning of scrapers and non-browser clients, since this tends to have lots of ugly side effects.
But regardless, I'm interpreting your last comment as saying that you should be entitled to behave as you please because you are a paying customer. If that's the case, then you are in for a rude awakening one day
The bot was also being generally courteous, only scraping the 2-3 pages needed to get the information for the human who spun up the LLM.
I don't think GP was at all suggesting he should be able to act however he wants, just because he's a paying customer.
I run an ecommerce site and I'm actually more worried about bots being rejected than accepted because of stuff like GP. We get a decent amount of business through ChatGPT.
To give a concrete example: I am a physician and have a site for my private clinic. I am not only unopposed to being scraped by bots/LLMs, I prefer it. The only value I gain from users being on my site is referral to my practice, and that same information being surfaced by search engines and LLMs just means I'm getting in front of more potential patients.
The point is that not all bots are bad. Assuming as much / implementing policies to that effect won't block all bots but it _will_ block the portion of bots that represent users considering giving you money.
I have a series of bots that monitor various eCom sites to monitor prices over time for big-ticket items I am considering as well as staples/groceries and everything in between. I have this little scrape/ingest pipeline because there's no other way to obtain this data... not even an API that I can pay for access to.
One of the large appliance sellers that I have in the scrape queue has gotten _hyper_ aggressive with bot detection to the point where even my personal head-full chrome instance doesn't always get to load the page. Guess who I will never buy that ~$2000 appliance from.
Significant sales (beyond for excess stock clearance/discontinued items) exist largely to generate some excitement from potential customers and to lure them in to taking a look at the site/store and to hopefully get them to buy some extra stuff that's less discounted while they're there.
The actual on-sale item is potentially losing money for the business or at break-even.
------
A person who's running a series of bots like this to track the exact lowest price on the internet for items over time and to always only buy at the lowest price it's ever been sold for....is probably the least likely kind of customer to buy anything extra that's actually profitable for the business.
Almost certainly! No eComm site wants the cautious shopper that's done their comparisons and knows price trends; they want the impulsive people with a line of credit. I doubt that my local grocery store is happy that I index all of my receipts so I know what I buy, how often and when the prices change so I know when a discount on something I buy every once in a while is actually a discount.
When I built this a few years ago, I had to write a lot of code to make all this work.
Now that's a simple "hey $agent, check the price of https://some-site.com/appliances/that-new-fridge.html and https://other-site.com/shop/appliances/that-new-fridge.aspx daily and notify me when it's on sale or has a promotion"
I would expect that more and more people will discover that their LLM of choice can do this, that their LLM of choice can help them comparison shop and do analysis of when - historically - is the best time to buy that new TV or whatever (yes, pricing on this stuff is seasonal!).
Maybe _most_ people will never do this, maybe I'm wrong. But if i'm not, blocking all bots "because bots are bad, mmmkay" is how you loose a sale.
Ignoring the specific price data example, more and more and more people _are_ browsing the web through an LLM. Assuming that continues, blocking bots is going to have the same effect on your traffic as doing everything in the "what not to do" section of the google SEO guide book.
I have a business where I clearly don’t want any bots. A retailer, maybe they do.
Not a rhetorical question, because most companies would welcome that interest from customers.
That's the thing; they can't, really.
Best we can do is compromise; allow bots and rate limit them. Maybe someday there will be a relatively universal standard for fetching pages akin to the `index.html/index.md` pattern that is emerging now. Put the bot filters on the .html path and some sort of micro transactions system on the .md endpoint.
In basically every cat/mouse game defense is forever one step behind...
USER AGENT
not user, an agent on behalf of the user.
the web was intended to be usable by agents belonging to users. in our path of the multiverse, the main agent we think of happens to be live interactive browser. but that need not have been, nor will it be in the future, the dominant USER AGENT.
for a glimpse at one possible futureverse, check out what the home assistant community is up to, how they assemble then update the ambient information displays on their walls. their USER AGENTS are doing what most HN-style "hackers" dreamed of reading scifi as kids.
the alternative is all your in home information owned by corpos when the info should be from user agents not corpo feeds. if you want to vote this idea down, you might be a corpo. :-)
https://news.ycombinator.com/item?id=32141027
Saying an API follows REST but doesn't implement HATEOAS (part of its definition), is like saying a database follows ACID but doesn't implement Atomicity. Or like saying a software is Open Source but doesn't allow use for commercial purposes.
Yet somehow for one of those 3 cases it became accepted to misuse the term that way ("[original meaning] but without this one constraint"), but if someone misuses the term for the other two cases then everyone suddenly cares and insist that people should use a different term to describe their thing.
The User Agent would get a HyperText document from a Web server, render it, and show it to the user. You could parse HTML, display images, and follow links - with the assumption the user would want to edit them like any other document. There was no encryption. There was no scripting for dynamic content. Styling was done via semantic markup. There wasn't even any POST when the User Agent term was initially applied. Everything was supposed to be like one big distributed document store instead of centralized hosting on few servers for many client devices.
All that is to say: Regardless what I think sounds nice myself, if the web was just the original vision & definitions instead of everything it changed and grew to become we wouldn't even be using it to have this conversation.
Interestingly, people seem glad to have telephony companies filter bots even though the original idea of the telephone did not call out robocallers as invalid clients. Probably because, unlike the modern Web, the average person gets a chance to see what it means to allow anything inbound even though the original concept never considered robocallers.
I agree with one aspect of the narrative you're saying, his idea was held as being for «documents». To be sure, document was more abstract than document on a Mac.
Either way, I'm very sad it mostly turned into animated brochures instead.
But the reason styles were separate, the reason everyone talked about semantics, was always about the universal knowledge DAG and enabling machines to help us navigate and retrieve it. It's why so many of the early agents referenced navigation (Spyglass, Navigator) or the interconnectedness of things (Mosaic).
So, ok, big document “store”, but that's a lifeless noun. The information you are looking for, the dots connected for you surfacing what you didn't know, that's a living growing thing, an agentic retrieval savvy knowledge engine, distributed for curation and resilience.
The librarian was always to be a machine or many, on our behalf.
Can we build other monetization? Maybe. There's certainly proposals. It's real hard when there's too many layers between the user and the output though. I suspect the solutions will be worse than what we have now. For now the answer is "just paywall", but given your invocation of "corpo" here I suspect that's not an outcome you'd be too keen on ;)
I would love some pointers to read up on this, what are some examples of this? Genuinely, not being a troll or ironic
There's a "look what I built!" thread every few days on the /r/homeassistant sub and the home assistant forums, too.
The data sources vary but it's usually a mix of devices that the user has locally and some remote data sources like weather/public transit schedules APIs or RSS and in some cases also utility rate info all integrated into a SPOG that is then displayed on some ambient screen (usually a hacked tablet or other cheap ARM device)
It was useless in 99.9% of the cases. You don't need or want to have constant access to how much free ram your computer currently has, as if that even meant anything.
So, after a few steps, that leads to the question "what is useful information? How do you define that?"
A basic attempt at it might be: useful information is something that causes you to make or change a previous decision. Take an action you wouldn't have otherwise done.
If you're living in the american southwest, and it's between the months of like, march and, dunno, septemeber, you probably don't need to know the current outside temperature to decide if you're wearing shorts or a jacket when you get up in the morning (you're wearing shorts because it's always hot)
Theoretically there could be some kind of "extreme" weather, aka rain, that causes you to do something different, bring an umbrella, but it's pretty difficult to not notice it currently raining, even if you're inside a house.
So what other information do you want? There's all sorts of stuff that's considered "news", but I'd argue that at least 98% of that is basically just entertainment. We'd need some kind of article or perhaps even a book to hammer out all the nuances involved, but seeing a headline in the morning that trump has caved in to iran again or killed some more random people on a boat, while sort of important to know, doesn't really change what you're going to be doing that day.
Knowing ICE is trying to do some domestic terrorism near where you might be could be useful information, but there's not a real good way to track that.
So yeah, what kind of information is actually useful to have in the morning? Stuff like calendar appointments would count, knowing you have things scheduled at specific moments or tasks to do, go grocery shopping, etc, todo lists in otherwords, but what else? That's basically all self-generated stuff by you about you, you don't really need any outside data sources.
So ... most people.
> That's basically all self-generated stuff by you about you, you don't really need any outside data sources.
Other than the ones you just enumerated that most people are likely to use, I need to know - using external sources:
- When to expect the garbage truck (do I put the recycle out tonight, too?)
- Is it a school holiday?
- When it's cheapest for me to charge the car up; if there are any "grid stress, please turn non-essentials off or we're getting rolling blackouts" alerts.
- Same goes for nearby fires or other issues that are going to change the "when do you open/close the windows and switch to AC" math.
- If it's likely to rain in the next few days so I change what plants get watered / when
- Packages that I am likely to get in the post today
- Weather (for display, not just for use in the automations mentioned above)
- Where my power provider is currently getting its power from (so I can calculate estimated CO2 emissions, if any)
And that's about 1/2, probably. Can't be bothered to go spend 20 min pouring through my HA config to build the full list.
I also stream the npr newscast as an MP3 as part of the "good morning, while you wait for coffee..." automations, if that counts for anything.
You mean that key whose only legitimate values also call every browser Mozilla? Arguing from the dictionary is so tedious. I'm going to scrape your site, but the idea that you have some obligation to like it because of the definition of the word "agent" is profoundly stupid.
A lot of people live off ads. I block ads. People who live off ads aren't obligated to like it - and are free to take any means to block it without moral hazard.
Identified by the user agent header. Which most bots fake or leave out to increase the chance of getting where they are not wanted.
Your bot is a good bot? Great, let us know when you've dealt with all the bad bots and we'll open the doors to the remaining (good) bots again.
https://www.ycombinator.com/legal
Just like you don't process raw data in a data warehouse or datalake by looking at tables, you use a SQL or BI tool, to process all the information on the internet you need to tool to digest it for you. For many, today that tool is a LLM chat interface or agent.
Google "How to make ham fried rice"; you'll get 10,000 articles, most pretty good recipes. But they're all different; most of them are just bait for ads. And most of them, the 10-line recipe is hidden between 50 useless paragraphs about how serving food is life's most important goal.
Now, ask an LLM to search for it, find the best combination, and list only the recipes. You get a perfect, 10-line recipe that doesn't waste your time.
I've also been known to include some ground up rocks in my meals. I'm pretty picky though about them, I mostly just want a couple specific varieties.
So why should I pick an LLM over google's inbuilt LLM or an actual cookbook on the desk? What wasted time is the LLM saving?
If you use google (sans AI), you're putting some trust in their page ranking algorithm. If you use it with AI, you're trusting the same algorithm (since that's how the model gets it's sources), but then you're trusting the model to evaluate the sources for credibility and extract the information you actually want.
No. Neither can be trusted as far as you can throw them. They're both incredibly invasive data brokers. Their customer facing products are just vehicles to show ads and Hoover up more PII and behavioral data on everyone.
Equating bot with bad actor in 2026 is Luddite behavior. Driving people to your site so you can serve them adspam or just make whatever operation they want to do 10x more difficult is the same.
MCP style APIs should eat the web. This doesn't mean the "open" web goes away. > 99% of the time I don't care for a domains particular FE at all. It's a complete waste not only of time, but resources and bloat. Pushing the contract into the agent should become good UX. Making things harder for good faith users should never be the goal.
And if not, what if it was a custom browser and not Chrome/Firefox/Edge?
They're just better at English now.
Like, all the comments of this post would go into my personal database simply because I loaded the page, and it would help me find old information I've read, and could also inform LLMs I use.
This should be built into browsers.
This is a "what if it's for your own good?" argument. What if I break into your house to clean your toilets? Although amusingly, if you're a "web alerts company" that I didn't contract and sends out alerts in batches on your own schedule, you will probably send a ton of your customers to my site at the same time and slashdot me off of the web entirely.
Scenario two: you use software to connect to their server and download a webpage. You are a "bot".
Make it make sense
This is a change. People put up new caching layers, and it urks people running a website in a single small machine. Yes, this could always happen with slashdotting but it's different now.
Are you implying that's a bad thing? Doing things for social status has been an integral part of society for millennia. It's a legitimate motivation that can benefit both the recipient of the status and the rest of society. In this specific case, if you assume that the author makes content that is useful to you only because of the social status reward, taking away that reward means the author will stop making content that is useful to you.
Your pontificating can't change the fact that he can do whatever he wants with his site.
I think you've hit the nail on the head here. I think that's a big reason why people want to ban bots.
For example, making a website that only works in Edge and refuses to load for anything else would be a shitty practice. You are free to do it. There's no one stopping you, but you should expect people to bitch about it and they might think you're kind of a jerk or maybe just bad at making websites. It's the same thing when a site actively rejects traffic from curl, wget, or any other common utility that many people find useful.
LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all. They do not display the page for a real person to read, they merely ingest its contents and then spit out something completely different.
No, not browsers, user agents (some of which would be browsers) and there was never an assumption of how a website would behave on the user's end, that's the job the user agent. All that HTML and CSS are only suggestions, but the power was always intended to be left to the user to decide if/how they wanted that data presented to them and it was always intended that the user be able to choose whatever tools they wanted to collect, process, and display content pulled down from the internet. That same principle is how we have ad-blockers. You are free to infest your website with ads, but as the person requesting the website I'm under zero obligation to display any part of that site I don't want.
> LLMs are not browsers, nor are they people, so I don't see why they should be part of this equation at all.
LLMs are just another tool used by people to collect and process the information available on websites. Maybe there is a distinction to be made between people using LLMs to get web content and corporations scraping websites to take training data, but even scraping has always been a common and expected practice. It's the current scale that is making things different.
I think I must be missing something here.
Overall systems and different business models need to be created that forces AI to pay it's fair share! Elon says there will be an abundance thanks to AI and we wont have to work -- ok how's that gonna work without different systems in place paying us?
I feel strongly about this topic and proposed some systems in this Substack post https://ryanspahn.substack.com/p/ai-to-pay-for-all-americans...
I have an old os (macos 10.11), running the highest firefox esr I can run, and I get denied by cloudflare.
But not always immediately - I get to enable javascript/cookies sometimes just to be denied.
they are not the folks we want gatekeeping the internet, they are opportunists increasing their OKRs
People trying to block bots end up keeping out all kinds of users. I get blocked frequently for using a regular browser, just with JS disabled. 99% of the time, I just close the browser tab and move on with my life.
And this is a complaint? I find nothing in this statement to sympathize with whatsoever. Isn't the contract of, essentially everything, that effort is required to obtain something worthwhile?
Please let me know where I'm getting this long-standing, fairly fundamental understanding of the world, wrong.
What's worked for me is I block any client that don't support brotli compression and http2. Seems to work well enough for stopping scrapers.
I didn't ask the not to use this site, my query was broad and complex and would take me days to do it myself. I wouldn't
I'm quite certain they earn a hearty commission off it, and I think it was mostly possible because the site was "friendly" to bots. Otherwise I probably wouldn't choose the site because it's never any of my top choices when I look for this myself.
So, maybe you monetise it like this? You asked, I answered. Doesn't fit every site or business profile.
I did that. I even printed and bound it in a bunch of dead trees and put it up for sale. The LLMs still stole it.
Yes, but no.
Also, using jargon that doesn't apply: a knee-jerk reaction is one that does way too much to address a small problem. Objectively, this is the literal opposite: targeted reactions to different aspects of a huge problem.
Substitute the word “website” for book, or training course, or documentary or published paper, or patent or one of hundreds of examples.
Now you see the problem.
If the the processing is subject to GDPR (e.g. if controller is in EU) then you do have recourse. You can complain to DPA or sue the company. The company is ultimately responsible for the decision to block you, at least in cases where you personally tried to access the site.
You are mistaking yourself, well your bot, as his target audience.
You might as well say “If I want to send you my commercial email, and you block it, you hurt me, the email user.”.
While your point of being concerned about cloudflair becoming a global arbiter of who gets in and who does not (which may at times not just mean blocking bots, intentionally or through technical issues), the need to block the deluge of bot traffic is very real for many sites and that is one of the easy options for them to deal with that. There are other methods like directives in robots.txt and nofollow attributes on links, but so many bots simply ignore those that they are not really useful.
> Bot traffic is not necessarily bad.
Nor is it necessarily wanted. In fact, it often isn't. Unfortunately practically all bot runners seem to either assume that their traffic is the special good kind or not care either way.
> I wanted the information and I have no way of getting at it without investing my own time and effort in going to your website.
I WANT! I WANT!! I WANT!!!
Well, that site runner wants you to access the site as a human, if at all, not via bots. Sorry to be the one to break it to you, but what you want isn't always the most important factor for the rest of us.
> but the real problem is that the website is poorly built and should be improved anyway
Firstly: just no.
Secondly: if you and your bot don't like our badly built sites, feel free to go get your information from those that you consider to be better built. Problem solved.
The article you're replying to describes in explicit detail how the bots and their operators have directly caused and continue to knowingly cause real harm to the author and others in similar positions, both financial costs and administrative/maintenance burdens that would not have otherwise been required.
You then respond "but if you block the bots then I won't be able to use the bots, and that harms me because I might have to read your web site myself..."
Are you serious?
> That might or might not be what you expected, but it's worth taking into account.
I would wager that for almost everyone who is blocking bots after getting functionally DDoSed by them this is absolutely an expected and desired outcome.
> And finally, something worth noting is that there are so many websites whose owners complain about bots, but the real problem is that the website is poorly built and should be improved anyway.
Both can be true. If you operate a git repository with a public-facing web interface for example there are going to be a lot of possible operations that are inherently expensive but also incredibly rarely used by normal users so it doesn't really matter, but the bots now ignore your robots.txt and are programmed to go after every link they can find, so they trigger every single possible expensive operation more times in a night than your actual users ever have in the history of the site while dividing requests across so many different IP addresses that rate limiting becomes impossible at the individual scale. These days they're even feeding the discovered URLs back in to their models to have them invent new possible URLs and trying those in hopes of finding content never publicly linked. They will send you thousands of requests for URLs that they literally made up.
Sometimes the site is in fact badly coded and operations that should be simple have higher costs due to bad design but you don't have to look very far to find situations where legitimately high-cost resources are exposed to the public because they're expected to be used in a non-abusive way. We should always be standing up against abuse of public resources, unless we want to lose them altogether.
> Bot traffic is not necessarily bad.
You are right, but whether it's good or bad more or less comes down to a cost/benefit analysis. As we've already covered infinite times, these bots being used to train LLMs cause significant real costs to the operators of these sites. What benefits do they offer in return? We know the clickthrough rates are terrible, so what other reasons would site operators have to make those real costs worth it? So someone can get a response back from an algorithm that confidently misinterprets or even entirely misreports what the data actually meant?
Even the most die-hard "information wants to be free" types who absolutely want their datasets trained on would probably prefer that the bots accessed the data directly via an API or downloaded a database dump rather than spidering and scraping a web interface intended for humans.
you literally can turn off cloudflare and use your own solution
The cure here isn't perfect, but much better than the disease of paying hundreds of dollars a month for scrapers which will never be beneficial.
Worse for the scrapers really isn't anyone's problem but the scrapers'.
Cloudflare is opt-in so I don't see that being an issue (yet).
If you can still buy a domain, publish a site, and other people read it as usual, then the web is still open imo.
But I can see a lot of negative network effects if/when Cloudflare gets to control 60%+ of web traffic.
They all sound as if they've been logic-twisted by some product idea they think is going to make them rich, and these blocks on bots are costing them access to the raw materials for their magnificently worthwhile project.
This is almost always the problem.
This is currently the problem that GitHub is having too. If they had remained with their crusty old rails architecture, very little of this mess [0] would be occurring right now. They would have been able to focus all their engineering talent on scaling the product rather than inventing elaborate client side state synchronization mechanisms.
[0]: https://www.githubstatus.com/incidents/qcvjkzcs7j74