>It also hasn't happened because there is a reward for cheating. Scammers will label their phone numbers as the customer service number for your cable operator or bank. Counterfeiters will label their batteries or phone cases as official accessories. It's hard enough to get people to correctly label their stuff—but fact-checking billions of labels created by millions of people, none of whom you know personally or whose identities you can verify? That's insurmountable.
There's more he says—the preceding paragraph is below—but that line "there is a reward for cheating" is something I keep thinking about. It's hard not to see in so many of the discussions of LLMs.
>For one thing, there are irreconcilable differences in the ways in which things should be described. A site run by transphobes isn't going to label every story about a trans woman as "a story about a woman," irrespective of what the standard says. Microsoft is never going to label Excel as equivalent to Apple's Numbers, Google Sheets, and LibreOffice's Calc. Hermès is never going to list a $40,000 handbag as being made of the same material and having the same capacity as a $30 leather handbag of the same size and shape.
When an LLM is the lens through which you view that text, you often don't have the context of where it came from, and the LLM is often really bad at filtering trustworthiness based on context when retrieving information to answer a query with.
I wouldn't go to a random site with a sketchy URL to get the customer service number for my xyz. I'd go to their official domain. Thus successfully navigating the treacherous waters scammers attempt to muddy.
You have an entire lifetime of experience and the wisdom of countless generations to inform you of the context with which you should process new information. You were specifically trained for years to be able to do this somewhat effectively. Doing the same for an LLM may or may not be possible. They contain much of that experience in some form, but are not well equipped to make use of it iteratively. It is incredibly expensive to attempt even a simple and ineffective version of it with our current methods.
If the LLM can achieve even 90% accuracy, it becomes more of a hindrance than a help. Do you make use of the function and trust it successfully understood the provenance of new information? If so, ten percent of the time you will be misled, which could be disastrous across a population. If you don't simply trust it, then do you painstakingly check the provenance of everything it tells you? Of course not, you already don't do that with current search engines and blogs and such. Yet falling somewhere in between would be so much worse than simply not having the LLM attempt to discern trustworthiness from providence in the first place.
But all of that is opinion and I may be incorrect. Perhaps someone will create some ingenious way of imparting wisdom and discernment to LLMs next year and this will all be trivial.
In principle AI Agents are superhuman at codeforces but tons of these vibe coded apps have ridiculously stupid performance problems.
The problem is a utilization of expertise thing, it can tell you with incredible detail about pros and cons and caveats to a problem if you ask about it. Truly superhumanly intelligent.
Calibration when it comes to actual decision making takes an incredible reduction in capability after that. Maybe it has something to do with the sycophancy or something but its like someone with exactly zero opinions (and therefore, taste).
My experience is that you can brainstorm with it on "Do A or B?" and it will produce a coinflip answer A, then if you continue and ask about a specific concern "what about this aspect of B?", it immediately 180s and flips back to B being way better than A.
Acting on this expertise when decision making points arise naturally for itself is where everything really nosedives. It's just gonna do whatever.
So when you are doing this "financial advice please" questioning, it is entirely hitting these weaknesses squarely in the face.
I don't think the average human has a particularly great sense of discernment either
The real lesson is this: trust in the source is the most important thing for 99.99% of what you read. You will sometimes read about facts that you can actually verify, but those are mostly useless in the grand scheme - except as a sanity check on the source. For everything else, you have to ultimately trust the source.
Now sure, sometimes you can get lucky and detect certain internal inconsistencies that prove that something can't be true. And to some extent, reading the same information from multiple sources you don't normally trust too much can be a slight signal that it might be true. But these are generally in the minority.
Again I am not ignoring reality. But for society to work and for us to live our daily lives with any semblance of sanity, we can't be expected to be fully informed on everything we see, touch, consume, etc. We have to be able to make reasonable assumptions about safety and accuracy or we can't function. And the time it takes to vet stuff is often a privilege (or you can brute force pay your way to safety).
I do not rigorously audit everything I read on wikipedia. I assume a certain baseline of accuracy up to a point, with of course exceptions. Do you pull up 3-4 high quality sources for everything you read or claim? Are you that academic about it?
Also, I didn't say you need to access high quality sources for everything, I again kind of said the opposite: you choose who to trust for what, and that's that. There is nothing more, and has never been anything more, that you can do. Sometimes people and sources you trust will still lie to you, or be wrong, and you'll know wrong information - this is also unavoidable.
Of course, the more important some information is to you, the more effort you should actually spend trying to verify with multiple trusted sources. Some weird fact about cats you saw in a TikTok? Fine to just believe it. Trying to figure out who to vote for, or if the elections were stolen? Probably worth checking a few more sources. Read some definition for a word on Wikipedia, or some old historical event? Probably fine. Reading about an even slightly controversial topic? Check the actual sources, don't trust the snapshot you happen to see*.
Of course, the way LLMs work is antithetical to this, as information provided directly by the LLM is not coming from any human source you can decide if you should trust or not. So you shouldn't trust it for anything important, unless you can verify. But for information or disinformation on the web, nothing has changed, except maybe the quantities.
*for a silly example: the exact definition of the word "electrocution" is a slightly controversial topic - whether electrocution refers to any (significant) injury caused by electricity, or only death. Pedants soemwaht regularly edit the article on Wikipedia to remove the "or serious injury" part of the definition, despite it carrying a link to the OED that includes this mention.
Yes, but if you read this "semantic" info directly from a site, you can judge the site and its potential biases, and decide for yourself "this sells Louis Vuitton, of course they will praise their handbags". Or to pick a more extreme example that nobody would ignore, if you somehow land on an extremist website spewing racist crap, your alarms will surely trigger by anything they say.
But through an LLM this gets whitewashed and presented as authoritative and "neutral", when it isn't. You might know not to trust the LLM either (if you're paying attention; we all have our unconscious biases) but I bet you most people will be caught off guard.
Billions have been spent on it.
Try searching Bing sometime. First result for Firefox is a malware clone site (you’ll have to scroll past the malware ad to see it).
This type of behavior is why gatekeeping institutions exist. We all pay a non-trivial cost because someone thinks that other people taking their lies seriously is somehow remarkable. It's not.
Research is hard work. Publishing fake research is just intentionally throwing wrenching into the machine of academia. All you've done is reinforce the need for gatekeepers (which never went away), show that some no-name journal isn't rigorous (they often aren't), and you've wasted a bunch of people's time demonstrating why institutions exist, when we know why they exist.
Genuinely terrible behavior.
How can this be fixed? Nothing prevents you from publishing straight BS, and LLMs will use any site that doesn't have a garbage reputation as a valid third-party source.
Of course, LLMs already use Reddit, Facebook and co. as a source, so it doesn't really matter...
As a human, this is fantastic.
> How can this be fixed? Nothing prevents you from publishing straight BS, and LLMs will use any site that doesn't have a garbage reputation as a valid third-party source.
Personally, I hope it can't, or the fix is onerous and kneecaps a lot of the utility of AIs.
I wonder if we can even have some adversarial competition, where competing businesses try to get AIs to ingest bad data to pollute their competitors' decision-making.
For starters: lets get some fake math papers out there. Given proof-checking systems, maybe they won't totally stop OpenAI, but hopefully the bad math could at least drive up the cost and slow AI down.
s/human/unscrupulous marketer/
Bad for humans in general, good for hucksters and con artists. Net negative for humanity.
> s/human/marketer/
I don't care where the bad data comes from. It's actually kinda good that some people will have a financial incentive to push it out and keep pushing.
> Bad for humans in general, good for hucksters and con artists. Net negative for humanity.
It's a sad state of affairs, but it's what has to be done. If we've got to burn the web down to save humanity from obsolescence, I'm for it.
The "net negative for humanity" was the attempt to build machines to replace human intellect. I'm not going to let any distaste for marketers or hucksters or con artists distract from that worse thing.
Why just web? Go straight for the powerplants.
Isn't the answer obvious?
Going for power-plants is stupid, as that would lead to unnecessary collateral damage. If you "go for" anything physical, it should be the AI data centers themselves.
But the problem with that is it would be illegal or incomplete. Publishing bad data is legal everywhere (e.g. even in North Korea, I don't think you'd go to jail for saying the US built a total of 3,234 cars in 2025, and the official statistics are a lie), and as long as the AI labs are chasing scaling laws, it could undermine their systems to some degree.
Furthermore, you are content to pretend that the rest of humanity is capable of doing this on a regular basis. And not just the top 10% of those who can read, interpret and weigh biases, but a solid portion of it.
It is a completely reasonable interpretation if you are an accelerationist living in the elite classes. In fact, perhaps, they are even counting on a majority of people thinking this is a fine environment.
I don't have the answers, but a lot of this is just a continuation of a very bad trend.
I think you misunderstand me: I want people to shit in the sea until it stinks so bad that everyone gets out of the water.
I'm not sure if we're talking about the same thing.
I think an acceptable good-but-not-perfect outcome it one where the "sea" (AI) stinks so bad that enough people get out that there's a viable life outside of it. The thing is people, individually and in aggregate, are lazy and competitive, so self-discipline isn't sufficient to sustain that life. There need to be problems that push people in that direction.
The web is foundational to AI, so that's a place where it's vulnerable.
Here I'm quoting from the website:
> It should go without saying that you should never publish fake content. While my experiment relied on tens of thousands of unverified stats, I ran it in order to share the results with the public, and the website, while still live, now carries a disclaimer.
"now carries a disclaimer" sure suggests it didn't, so "you should never publish fake content" really doesn't seem like a standard the author actually held themself to.
The world isn't perfect, man. You have to pick the least bad option. Humanity permanently ceding control and understanding to machines is worse than an internet full of garbage.
And the logic of our society means the former is inevitable if the machines are capable, so the machines need to be made as incapable as possible.
We're in "something must be done; this is something; therefore it must be done" territory here except usually the something in question is merely useless and not directly harmful to bystanders in the way this proposal is.
https://talyarkoni.org/blog/2018/10/02/no-its-not-the-incent...
Good behavior that facilitates bad behavior is not, in fact, good behavior. For instance: it's not "good behavior" to hold the door open for gunman hell-bent on destruction.
The AI labs aren't building AI to help humanity, they're building AI to win at capitalism. Which a big reason why the AI is coming for the intellectual, creative labor that people often enjoy, before it comes from the dirty, unenjoyable labor.
Regardless of how the data is sourced, this is bad. It's poisoning the internet.
Besides, such issues come from a lack of due diligence, which have only been exacerbated by AI.
I'm very well aware of that. Unfortunately, this technology was invented, and the logic of our competitive society leaves little chance to contain it except scorched earth tactics.
You can find people complaining that Google search results decayed to being unusable prior to LLMs being a big thing. The marketers and spammers effectively won that fight. They'll presumably win this one too.
LLMs can be trained on arbitrary token strings, which semantically contradict each other, because semantics is not involved.
The way to attack the "garbage out" problem with an LLM (like anything else) is to address the "garbage in" problem.
You will get better quality out of an LLM that is trained on nothing but mountains of unvarnished truth: logically sound propositions and true facts about the world.
But even then, interpolating among such is not reliable! We can't add together two different truths and divide by two to obtain a midpoint truth: that makes no sense.
That is still a sort of "garbage in" problem. But that one is not in the data; it is in the design of the processing concept. The body of algorithms that power LLMs have a fundamental garbage aspect to them that cannot be removed; only a replacement of the LLM with something different can address it.
So hopefully most of these statistics move towards first party sources.
People stop trusting what they read on the internet.
All alternates are just attempting to improve the baseline trustworthiness of random information from the internet (and/or LLM that's baby-birding it back to you). Any increase in how much people blindly trust random stuff from the internet only serves to increase the blast radius when it is wrong.
It's a ton of effort and expense, but probably feasible from a technical point of view. So instead, the model providers will probably do something silly like wire up rules like "NYT=Reliable;my.blog.forbes.com=Unreliable" and they'll say the issue is solved.
Is this information on this website true/high quality?
We need a google for LLMs!
Oh, but LLMs don't consume ads.
We need ads for LLMs!
Help.
Google came and worked wonderfully for a year or two, until people learned to game page rank algorithm. Soon first google pages were just SEO spam and now half of the google is just spam, big corp websites, etc.
All of that precious stuff from small hobby websites about cool things to blogs that used to be wonderful read like https://www.joelonsoftware.com.
AI were not hurt that much by spam, I guess there were at least some curation done, the question is if this curation will work and for how long. It seems we are throwing the towel here as in case of search.
Backlink ranking was/is a reliable heuristic to answer that question across virtually all topics.
So I disagree that the "problem was not about quality."
Spam/scammers will always be in whatever landscape there is, and I agree part of what google does it keep them at bay.
Trap Street - https://news.ycombinator.com/item?id=35768306 - May 2023 (29 comments)
Trap Street - https://news.ycombinator.com/item?id=17738847 - Aug 2018 (21 comments)
Trap street - https://news.ycombinator.com/item?id=4992748 - Jan 2013 (22 comments)
South Pacific Sandy Island 'proven not to exist' - https://news.ycombinator.com/item?id=4818584 - Nov 2012 (46 comments)
Not a Word: The story of Lillian Virginia Mountweazel - https://news.ycombinator.com/item?id=1504126 - July 2010 (10 comments)
Other than some sort of social punishment I don’t see the cost for the copier. For the original author there is a cost of making your map less accurate on purpose.
So I think to answer my question more directly, yes, it's just copyright laws.
My point, before understanding that, was that from the map consumer's perspective it doesnt make a difference. And from the original creator's perspective it has a little accuracy cost. So without an additional mechanism I didn't see the disincentive for copiers. So I wondered if it was some ancient form of copyright law or just strong sense of morals or what.
There were a few factors discouraging mapmakers from blatantly copying each other:
Rulers granted printing monopolies from the late 1400s. A mapmaker like Ortelius would print "cum privilegio" (lol) on a map. If you copied it, your expensive copper plates could be seized, along with a hefty fine.
Spain and Portugal each had a master chart which were state secrets because they held navigation knowledge worth a fortune. Portugal handed out severe punishments for leaking charts. (Leaks still happened: the Cantino planisphere was smuggled from Lisbon to Italy in 1502.)
Booksellers' guilds kept registers of who owned which work, and enforced it.
And lastly, copying meant hand-engraving a new copper plate, which took months of work and real skill. The practical barriers alone got rid of most of the copiers, but it didn't stop the big houses.
Some followup links for the curious: https://gist.github.com/shawwn/b85668a421aa443879b15aedd10c5...
(I used Claude to research this fascinating rabbit hole, so I'm basically paraphrasing here. But this comment is my own original writing, so treat it as you like.)
They now train on their own content.
Eating your own d̶o̶g̶ f̶o̶o̶d̶ poo.
Now, the model slips it in without any verification and casually builds on top of it.
Even if you are in the minority who asks for sources and truly verifies them, they are often made up (don’t exist) or don’t say what the LLM said they said.
1. This is a 2022 era picture that LLMs just suck up what's on the Internet and regurgitate it. It hasn't been true for a long time.
2. There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable.
The other day for me Google search AI cited triggerbites which is a vibe data website.
The LLM unless explicitly prompted cannot distinguish the validity of a data source.
When I ask ChatGPT certain questions, it very clearly shows me the sources it's searching through in the form of favicons, and they're always the favicons of sites that are in the first 5-10 results for a Google search query that would fit what I've asked the LLM. It will then cite those websites as it's source, with a direct link to the page that also shows up in an equivalent Google search. It does that daily.
On the other hand, I recently asked ChatGPT for some information about a group of people, and the headshots it put with each summary were very clearly not the people it was describing. So hallucinations do still occur even on the latest models, it just seems unlikely that the favicons would be LLM-generated.
if anything it’s more true now, AI chatbots rarely use raw LLM output, instead they have CoT and use search tool calls. and the speed at which webpages are crawled is in the matter of hours. I posted something here on hn, then searched for it on google 1 hr later and its AI answered with my comment
https://share.google/aimode/PkQUVfA68AlHGb7Yr
there’s several companies right now making a killing of organic placement on chatbots similar to the SEO rush of early google
> 2) There's almost no one who has a stronger incentive to distinguish LLM generated text from non-generated LLM text. Training classifiers is pretty manageable
i think it’s pretty naive to think that if LLM companies would develop LLM detection they would use it to skip training on that content.
instead, they use it to train their output to be less detectable.
LLM detection is not solvable because it is adversarial, an arms race
There's a tension there: they want you to use AI for everything, and if they get what they want then new information will be embedded in LLM text.
> Training classifiers is pretty manageable.
Hopefully such things can be opened up, so the slop can be iterated on until it passes. It doesn't need to be perfect, just enough so more garbage gets through.
In the long term, this is probably a real concern that they have to tackle. They'll have to move away from just classifying things as LLM generated or not, to attempting to judge the underlying quality, and that's not an easy problem.
It's both a blessing and a curse, especially if I'm wrong. Sometimes the Google AI misappropriates someone else'se quotes to me, not sure why, though those are usually if I ask it about myself directly and it tries to generate a summary of my HN post history.
On one other occasion... it was a post I had posted 2 weeks prior, I was trying to do further research into the subject.
LLMs are quite good at "I need a thing that does [vague description]" searches, FAR better than ~all "semantic" searches. But oh boy. I do not like them "answering" queries. Feels extremely dangerous.
Related: Preserving Historical Truth: Detecting Historical Revisionism in Large Language Models
A lot of "facts" get created this way. You end up with hundreds of citations, but nobody seems to know (or care about) the original source. And meanwhile, freelance militant conformists (rarely shills, usually people intensely addicted to updoots) use those citations to personally attack people who cite primary sources, and to get those sources banned if possible.
edit: Shills are ancient technology. All you need now a fleet of repost accounts that exist to updoot fools. The fools, having the happiest moment of their lives, then double and triple down, becoming more and more gleefully hostile. That's when you finally have to use a shill to get them to open a patreon. Then you can get the world out of them for a couple grand a month.
That said, I expect that OpenAI will eventually have its own search infrastructure that rivals Google's, and the overall system will get smarter. Whatever Google did to recognize this site as trash, OpenAI will do as well.
"History shows that when a slope appears, regulation steps in, technology evolves to solve the problem, or the culture shifts to reinterpret the tool. In almost every case, the feared bottom of the slope was never reached because humans constantly built ramps or bridges along the way."
Hard disagree. Sometimes it's avoided, any other time we just go into cope mode. We discover on the ride down that the bottom isn't so bad. Turns out our collective control mechanisms didn't fail ourselves if the initial assessment was wrong.
I wonder if similar to how many engineers now write 90% or maybe 99% of code with AI and primarily weigh in on the plan/design, will writers too primarily weigh in only on the content brief/plan and then the actual post is seldom read in full (once you have faith that the AI is not making up citations). Because the audience of most writing is now bots prose and style are no longer relevant.
It's an excellent way of showing how unreliable the information you find online is (and it will only get worse in the future), but willfully creating misinformation and made up shit and offering it as fact feels hostile to humanity.
Surely nobody got hurt as a consequence of those "facts", but bad decisions with real world impact could be made regardless. It feels like the behavior should be shunned in a general case (the same way I'd shun those numerous spam sites that pollute search results and very much would prefer they not be allowed to exist), even if in this case the intent was good.
Problem is, you can't really do anything about it, other than figuring out a way to take down hostile sites and keep them from popping up, so it's even worse than fraud or even just bad research in academia.
Except for the fine print. LLMs are to be used for entertainment purposes. lol
If they didn't, they'd be unable to compete with their competition and also we'd get worse models that can't reason about many domains or do that much due to limited training data/generalization. I don't think we can put the cat back in the bag or mandate only non-problematic training codebases without shooting ourselves in the foot.
Now, if those AI orgs were mandated to properly compensate everyone whose data they've mined, to track which neurons get invoked and to pay accordingly based on what was used to train them... ehh, that's about as unrealistic as stopping. Kind of a bad situation all around, unless you abandon the idea of any sort of copyright.
It's interesting to have a specific case where obviously false data is being spread, but there's nearly an infinite number of more subtle variations around.
if you just lie and an llm or a newspaper or a kid believe you, at that point it isn’t a criticism of their naivete, you are just lying
similarly hacking through systems or picking locks isn’t a showcase of weakness if systems, it’s just being a hacker
good thing nobody falsified statistic just for the sake of it -- so being a sabotager and a foreign spy was always part of it too.
Even when there is no evidence to support something, and no motivation for anyone to promote the falsehood, models can build an ediface out of false claims.
There are always people prepared to invent a story to explain why something is true. Just so they can sound knowledgeable. They aren't trying to perpetuate a lie, they are just under the same misconception as others, quite a lot of them will believe their invented explanation even as they say it.
It's going to be a difficult task making an AI pick the correct one of two mutually exclusive pieces of evidence, especially after it has found one piece and does not yet know the other exists.