Creepy Crawlies
55 points by zdw 22 hours ago | 10 comments

chuckadams 26 minutes ago
Given the nature of git, wouldn't all that HTML be highly cacheable? I get that's not free either, but it's got to be a lot less intensive than having cgit generate it every single time.
reply
kees99 45 seconds ago
Or, html generation could be very, very fast in the first place. For example, serving static files is usually just as fast as a cache.

Interestingly enough, cgit bills itself as "a hyperfast web frontend for git repositories" [0]. Alas, nothing is fast enough for the bots, it would appear.

[0] https://git.zx2c4.com/cgit/about/

reply
Zariel 23 minutes ago
That was my first thought, varnish (vinyl these days) in front of the website should dramatically reduce this as the git repo should be practically static for most of the content.
reply
teo_zero 19 minutes ago
But there are "cubic bazillions" of possible URLs that are being requested. Even if they boil down to "only" some millions actual commits, their rendered HTMLs are all different.
reply
initramfs 29 minutes ago
I've been noticing page views in the past several months with a much wider span of origin on my Blogger stats. Before I would get a few from several countries, but now I am getting views from tiny countries and obscure or outdated browsers and operating systems, which leads me to think scrapers could be using VPN services in various countries along with header anonymizers that mask the device that they are using.

Extensions like ModHeader, BrowserMask do this: https://github.com/apify/crawlee-python

https://github.com/mthcht/Masquerade-Spoofer

Great for AI scrapers, bad for hosters and everyone else.

reply
Artoooooor 34 minutes ago
How expensive would AI access be if every user paid their fair share instead of shoving it on the people doing the actual work?
reply
parineum 8 minutes ago
You mean shoving it onto the investors?
reply
jruohonen 22 hours ago
Off-topic, but anyone with which he did the plots?
reply
dingaling911 41 minutes ago
reply
jruohonen 38 minutes ago
Thanks, and, yes, I don't write with LLMs, as seen above.
reply