The main problem on my site is that bots hit the CGIT repo, where visiting files triggers syntax highlighting scripts.
I put in a skill-testing-question type system where you get a maximally long-lived access cookie if you answer right (so legit users are bothered as infrequently as possible).
* I wanted an elixir version so I could run it alongside my existing apps instead of having it sit as a proxy in front of them. This way my existing apps can validate if the caller is already whitelisted and if not redirect to the challenger https://sphinx.8by3.com which sends you back to a site if you pass eg. https://tuhat.net
https://www.sammystraus.com/#using-llms-to-make-the-emacs-we...
I experienced the same thing after submitting articles from my small photography website. I did debate using some of the off-the-shelf solutions like Cloudflare, Anubis, or iocaine, but I ended up building my own L7 proxy that utterly fucks the bots up: https://robertmay.photography/journal/bwbach-my-guardian-gob.... I will make it generally available at some point once I get it optimised how I'd like, but currently I'm able to trap about 3000 concurrent crawlers in my tarpit of slow requests on two small Hetzner VMs. And that 3k is really just the limits of the networks hitting my servers right now, I think it'll go significantly higher than that.
I haven't really seen anyone else combat them this way, possibly because many of the standard proxies and servers are set up for fast requests, not serving them as slowly as possible. Deliberately serving page loads for as long as the requester will accept packets turns out to be a really funny way to stop the botnets - when I just rejected the requests they in turn hammered my server even harder. By tanking the request volume with a ridiculous BEAM-based proxy you can reduce their request speed to a crawl and I'm fairly certain a couple of them have blacklisted my server now.
EDIT: sorry, that was a dumb question before reading your post. You actually learn the timeout they tolerate. Nice!
Some people liked my work, found me on LinkedIn and sent a connection with a positive message. Other people tried every which way to SQL inject or attack the site, posted slurs in every available input, etc.
Someone copied the site and hosted their own version with a couple minor changes. I only found that one because they had left one of the links in, which gave me referring traffic from their copy.
All that from one HN comment, very much pre-AI too, not to mention the standard daily slew of requests probing for what I assume are known vulnerabilities in Wordpress or PHP.
The internet is a floodgate of different actors, many of which are automated, and a link posted somewhere like HN opens the valve especially wide.
https://pagespeed.web.dev/analysis/https-sprinklz-io/bmc4mv1...
Just merely existing will get you scanned these days.
Who is?!
It’s really starting to suck. The worst part is these are coming from consumer vpn providers that we can’t just block (lots of real users paying real money on our service are vpn users) or they come from big residential isps. The whole residential relay thing is a hard adversary, since it’s spread across so many real users paying ips that you can’t whackamole it. I’d say 1% of residential ips are participating in a botnet (there’s prob some real numbers out there if I cared to look).
This is just going to push everyone harder into walled gardens. Web input biggest inbound but our content is our mote, we are going to have to turn off non logged in content viewing, which will have real business impacts.
Not even third party login worked (Google, Linkedin, ...) to curb this.
They were logging in, scraping the client API for vulnerabilities/possibilities and then spammed erotic dog poems to Claude Fable until the $5 welcome credit got drained.
Fun New World.
That sort of behavior would justify a name and shame imo
Inconsiderate shitty services don’t deserve the politeness of protecting their name
Though they probably use the cheapest possible models they can so the damage from their dumb business model would be limited
At first, I assumed this was a single blog post, especially with a "Table of Contents" at the top. I guess it's following some old-school blogging conventions, but AFAICR, even those sites retained individual pages for separate posts.
>So I took a look at some data. I added top-level logging for every endpoint (how many times was a request made to each url path), and I found that a majority of the requests did not even make an attempt to fetch news articles! For reference, my website is built as a single-page-application, and what this means is that a majority of the requests did not even run the javascript necessary to render the page (!).
So you got hammered by curl requests! But isn't that part of your delivery static anyway? Were you regenerating the same HTML over and over?
>many, many requests per minute?
How many was it?
Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
But hey. Silver lining: free load testing! ;)
i dont think this was the cause of the slowdown, it was just sort of the thing that led to the discovery that this was mostly bots.
in general, because the purpose of the website is to allow users to generate news feeds based on custom criteria, it does do a bit of meaningful work on each request (ones that do follow through and load the full page).
> How many was it? > Also interesting that it was a single "user". (user-agent?) Did it come from a different IP every time?
I'm intentionally leaving some of the details vague here, both because I don't want to be too harsh on the original author (it may have just been accidental/a buggy program/etc.), and because I don't want the traffic generators to know how im flagging them.
That's a big reason that I never post any of my apps here.
Welcome to hell, kid...
On the other hand, “multiple times per minute” doesn’t seem that high? Even if it was 60x per min that’s fully within the limits of what a 2015 laptop in the closet can handle.
HN has become unbearably AI slop oriented over the last couple of years. It’s nice to see something else.
IMO the important thing to try to crack down on is vote manipulation; if you have that under control, then you can probably happily allow self-promotion and trust that only the interesting stuff will rise to the top.
Perhaps a contribution requirement per post? Though this proposal just invites bots to actually post comments because that becomes a requirement for posting links.
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
https://news.ycombinator.com/newsguidelines.html
I assume that users who primarily/only post their own stuff and get flagged a lot reap the consequences, there's plenty of spam filtering on here thank goodness.
> Please don't use HN primarily for promotion. It's ok to post your own stuff part of the time, but the primary use of the site should be for curiosity.
As per the wiki page for the C10k problem[1] where it says "_was_ [emphasis mine] the problem of optimizing computer networking stacks to handle a large number of clients at the same time". This was solved over a decade ago...
Now get off my lawn!
(TBF I think the author acknowledges this)
You can do that with any system made in the past 15, maybe 20 years.
From the wikipedia page:
"As of 2025, the problem has long since been solved, with the number of possible connections to a single computer being in the millions."
I banned Alibaba's Cloud two years ago and it still hits one of my domains 1 million times per day, despite every route 4xx-ing during that time.
Even Apple has aggressively ramped up its scraping in the past few months, but thankfully they do respect robots.txt.
It's frustrating because some of my real users route through these clouds via their work VPNs. Also, there are a lot of less-than-reputable AI scrapers using global bot-nets via residential IPs to scrape, making it even harder to detect and ban them without affecting real people.
* like leaked .env routes, brute forcing admin panels, and so on
You may as well silently drop the packets if they don't adapt.
Google DNS are busy but its a hand full of IPs and reasonable. Cloudflare is non-stop never ending traffic. I don't know WTF they are trying to get. I've also seen TCP queries from those same Cloudflare IPs so this is not the result of a DDOS attack on them.
1. Breach router or desktop that hasn't been patched in ages via mass script
2. Establish C&C using cloudflare tunnels
3. Sell these hacked systems as residential proxy endpoints to everybody from state actors to teenagers
4. Some of them will be data scraping services
You are chasing your tail trying to ban by ASN or IP. It may work for another couple of months, at best.
Similar to ad click fraud and scam ads, the cloud provider gets paid, why would they turn down the money?
Blocking cloud providers is strictly for us dirty self hosters who are sick of our logs becoming unreadable and our visitor metrics useless from all this malicious traffic.
Why allow random requests coming from the clouds?
However, they could just use residential proxy botnets.
I believe the future is static resources with dynamic apps being only offered to people who can authenticate. And even then I’d recommend grandfathered accounts (eg telegram users that had an account for at least a year).
Agreed
> I believe the future is static resources with dynamic apps being only offered to people who can authenticate.
I don't know about static resources, but I do think that identity will come into play.
Why allow random requests coming from the clouds?
However, they could just use residential proxy botnets.
Traditionally cybercrimes have been prosecuted by the victim going to their local PD (or the FBI) with the IP address of the offender, and then law enforcement subpoenas the ISP associated with that IP address to figure out the customer associated with that account, and then they have a case for prosecution. It all works as long as cybercriminals are less numerous than law enforcement, fall in the same jurisdiction (or in jurisdictions that have reciprocal extradition treaties), and have a real identity that can be tied to their online activities.
Now, many law enforcement agencies are tied up prosecuting political opponents of their bosses. To the extent that they have bandwidth to investigate and prosecute ordinary crimes, they're going to prioritize the ones that target a.) the government that backs them (of which there are many - you see another headline about the government being hacked every week or so on HN) b.) major corporations that have sophisticated IT systems that can aid prosecution or c.) headlines that will get their boss a PR boost and themselves a promotion. Your personal website does not register among those priorities. You probably can't even prove damages from the intrusion itself. The perpetrator might be a Russian bot that the U.S. doesn't have jurisdiction over, or a Canadian teen that the U.S. previously might've been able to prosecute with reciprocal judicial treaties but now has no leverage over because we've blown up our relationship with Canada. It may be legally a crime, but economically it's not worth prosecution.
Traditionally security researchers had to be very careful because they were doing everything above-board, with their real identity attached to their actions, and they made a very attractive (and PR-worthy) target. Somebody who's port-scanning from Russia through a VPN or residential proxy network is a.) anonymous b.) untraceable c.) out of U.S. jurisdiction and d.) not a headline-worthy target anyway.
A couple years ago I predicted here that a lot of things will break as the U.S. transitions from a high-trust to a low-trust society. This is one of them. Much of our justice system works well when criminals are rare and utterly falls apart when they're ubiquitous.