Thing is, though, verification is knowledge itself. And if we can get to a point where the important questions have answers, we can surely have the verifiers explain those to us in ways we can understand them.
This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.
I think this is an artefact of how we train them, and they could be trained to be better at this if we focused on it, but at the same time, it shows its possible for these things to be able to do things they cant explain to us (granted, they can't explain them well - you can bash your way to understanding with them anyway but they aren't good at it)
It's amazing to me that language models are still such poor writers and communicators. The mathematics community wrestled with this recently, when many solutions dropped into their laps but were not explained very well...
Thing is, (1) generic LLMs will improve, and (2) their lack of ability in this area has a lot to do with the rhetorical and structural habits they pick up during RLHF/instruction-tuning, which trained them for general-purpose comms. Zero-training base models can presumably be trained specifically to communicate scientific concepts. Interesting times.
(Edited to add: With respect to (1), compare Opus 5 to 5.5. 5 was a terrible communicator, even of software/coding concepts, whereas 5.5 is excellent and leaves very little to be desired.)
Many people don't like the style, probably by association with slop and fatigue more than the style being fundamentally wrong.
Being a good communicator is an entirely different thing, you can have terrible spelling and grammar and still be good at getting ideas across. It is a common situation when the communicator in question is not using his native language.
That LLMs are so good at writing and so bad at communicating is not surprizing. The key to good communication is getting into the other people's mind, LLMs can't do that, because they "think" in a way that is completely alien to us. We use words to express our thoughts, thoughts that are based on a physical world model: touch, movement, sound, light, etc... LLMs have no idea about how the physical world work, everything the see is just words, they have no intuitive idea of, say, objects falling down, just that "falling" and "down" go well together. And for explaining things, they will go "someone asks for an explanation, explanations usually use these words, I will be using the same kind of words".
Hm, that may have to do with (faking) confidence. I've often enough with 5.5 still run into cases where it confidently explained something, at the surface it kinda made sense, but the more I dug and verified, the more it became apparent that it was entirely wrong with the most crucial core concepts. It has gotten to a point where you really need a domain expert though to realize this, which makes it tough for using it in areas where you are not an expert and don't have easy access to one either.
Verification isn't knowledge. Far from it. You can verify statements without knowing why they're true. Variants of that concept are e.g. used in cryptology.
You need humans to truly understand (to know the context) in order to make correct judgements. Without actually knowing the context, you have no way to compare.
The same goes for our own time. A powerful AI can tell us things at the edge of what we can grasp -- or beyond what we can grasp -- and it would nevertheless be knowledge that we possess. We can work to verify it, we can have it explained to us as though we were schoolchildren, or we can simply accept it as a static fact of the natural world.
In truth, we're going to have to learn to coexist with minds that are more powerful than ours. At least in some respects, if not in all respects. The good scenario is that they share information with us, and we receive it however we can. "Head in the sand" and "mutual disdain" are bad outcomes.
You might want to look up the actual definition of knowledge. It's a pretty interesting topic in epistemology really, far from your simplistic take.
You already live in coexistence with minds more powerful than yours. Society works in a way that you never need to notice that, those minds taking care of things beyond your grasp for you.
The downside of that "convenience" is you developing an over-simplified bubble-view of the world, rendering you prone to arbitrarily hazardous misjudgements.
Here, you align foolishly with interests that cannot possibly be your own. Namely, exploiting scientific knowledge for personalized monetary gain on an unsustainable scale.
When a tiny group of people hogs essentially all scientific knowledge, that renders you what?
Your idea of being graciously provided with some "basic income" or whatever is blind copium. You would live as livestock at best, to be discarded as useless whenever your masters see fit.
As far as I know labs try to train the other way, else we'd have monstrously long and useless outputs, but its hard to overcome these natural biases for length
Like a superposition of all kinds of writing styles, all perspectives on every topic and all kinds of ways to do things right and wrong.
In post-training (and of course already by sensibly curating the input data in the first place), people try to select useful submanifolds from that.
Ideally, you get a combination of all the extant strengths of people combined into one model. Self-improvement then leaves the sphere of human capability for good.
And the abstractions do still matter, since in many cases it is an issue of tractability. Perhaps an LLM can force its way to a solution through 3 million mind numbingly boring and convoluted steps, but if the number of steps increases exponentially without the right abstraction that means it can only handle a slightly larger problem than a human with pen and paper.
> https://en.wikipedia.org/wiki/Hierarchy_of_the_sciences
First math and CS, where formal logic dominate, and AI can check/verify its own answers very quickly. Then physics, downstream of math. Then materials science, which in some cases is practically a branch of physics. Then chemistry more generally. Then biology. Then various social sciences.
Fields medalist and as Tao listed as one of the most important mathematicians of our age.
Scholz is known to find „precisely the correct and cleanest way to incorporate all the previously done work and find an elegant formulation for that — and then, because he found really the correct framework, go way beyond the known results“
In a way that man is the complement of an AI, he generates understanding rather than just proofs.
Maybe he is the right template for the mathematicians of the future.
https://www.quantamagazine.org/peter-scholze-and-the-future-...
See the Liquid tensor experiment:
https://xenaproject.wordpress.com/2020/12/05/liquid-tensor-e...
https://www.ma.imperial.ac.uk/~buzzard/xena/pdfs/liquid_tens...
" I spent much of 2019 obsessed with the proof of this theorem, almost getting crazy over it. In the end, we were able to get an argument pinned down on paper, but I think nobody else has dared to look at the details of this, and so I still have some small lingering doubts. "
In a sense, mathematics is not very different from software engineering.
More precisely: Curry–Howard correspondence
https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...
One bit stands out to me: to better understand, Scholz will work the problem backwards. Sometimes that strategy is just so spot on.
--
To my eternal shame, philosophy (like this OC) makes my head hurt. Pretzels my brain, ouroboros like (aka "head up my own butt").
Caveats covered, here's my cave man cold take on two aspects of this topic.
#1 As we're all trying to paraphrase here: answers are great; understanding is even better. I like the word "legibility" (h/t James Scott). The current models are not very legible.
Which directly leads to:
#2 Understanding enables predictions.
If LLMs are stochastic parrots, then pi.dev (GPT + reasoning + ralph loop) is a drunken sailors walk (thru a solution space). A kind of messy heuristic for optimization. With some kind of gate for declaring "done!" Pretty useful.
However, I don't expect Pi.dev to make predictions. That'd require something else. Maybe that something is being worked on; am noob, so I wouldn't know. (I'm still unclear about whatever Jeff Hawkins is doing, in practical terms.)
--
Wrapping up: Automated prediction would be cool. That's very different from finding solutions to problems. And the current cohort of AI can't even explain why, much less make predictions.
Assuming predictions requires understanding. Maybe there are other strategies to generate predictions. Like a meta drunken sailor's walk, with drunken sailors all the way down.
(Someone please correct me as needed. TIA.)
[Aside: That article has an ad for their podcast They Joy of ₍₎. Perfect. I have no idea if it lives up to its promise.]
However, even getting truth answers will soon be beyond AI. It is of course making impacts in Math and in publishing in "data heavy" subjects where you can just formally verify, program, and/or munge data.
But in "real science" where we actually need to build a lab, buy supplies, hire personnel, get approval, form physical test harnesses, install sensors in proper locations, calibrate everything, then finally make one measurement, question the whole thing, then finally make a conclusion, AI is of limited benefit.
Yes, it can help speed up coming up with some formulas, spitballing, or checking existing research. It can even help write the jank code for sensor calibration, measurement taking, etc. That's not nothing, but we are _very far_ from meaningful contributions to real experimental hard sciences.
As for math, theoretical physics, computer science, and other "logic based" studies, the question becomes one of benefit to cost of true understanding versus simply knowing something is true.
I'd argue that it paints a starkly negative picture of most science. Once answers are easy to obtain, it seems most people are concluding that actually understanding why its true isn't worth the effort, which means that the question maybe wasn't worth asking.
Its the science equivalent to the recent realization in tech: most ideas suck, your product didn't need more bells and whistles. Actually the engineers pushing back on every shower thought from product was a good thing. The friction that stops you from putting every random-ass feature into your already bloated useless SaaS product was good, and now the sales guy is pushing features to prod for a customer demo. We thought "too many ideas, not enough engineers", we now know "not enough meaningful quality ideas".
In the same way we thought "too many questions, not enough scientists", we now know "not enough real study on meaningful topics".
Knowing that something is true or false will at least stop humans wasting time trying to prove or disprove it, and allow them to find a more elegant proof, or investigate the implications of the true/false result.
Autonomous or human-directed AI agents submit their work to a public repository.
Then the scientific community draws from this pool of results, identifies the important ones, and starts working on adding the results to the greater body of scientific knowledge. This work of interpretation, synthesis and question formation should be recognised and rewarded in peer-reviewed academic journals.
I wrote a blog here talking about it: https://colinocallaghan.com/posts/answers-are-getting-cheape...
Is this description AKA. theory vs praxis? Academia vs Engineering?
I propose a two-part approach.
First, autonomous or human-directed AI agents submit their work to a public repository.
Second, the scientific community draws from this pool of results, identifies the important ones and asks the next set of important questions. This work of interpretation, synthesis and question formation should be recognised and rewarded in peer-reviewed academic journals.
I wonder if this dichotomy fully captures the value of scientific discoveries. One could argue that their practical impact can be independent of whether humans fully understand how they were reached.
See: most of pharma
A good word predictor happens to be good with closed systems with very defined rules for which a huge training corpus exists and we are suddendly freaking out about how it's gonna become so much better than us at everything that we are basically obsolete.
When in doubt about where AI stands, ask it to create a poem, a joke or a riddle. And see how little actual intelligence it has.
If you're in academia, you've probably noticed a major shift recently. Research production at an industrial scale through AI.
Grants have "always" been awarded to professors based on quantity multiplied by quality. Now, some professors are churning out 1–3 papers a month with the same group of co-authors. All the grant money goes to them. Grant givers are just only counting the numbers of papers being published.
The Industrial Revolution has hit academia, and some academics are busy arguing for the artisanal value of research while some are creating factories for research.
The problem is the systemic disincentivation of true science, the pursuit of truth and knowledge unencumbered by greed.
When you speak favorably of "factories for research", you adopt the position of a mindless consumer, relishing the prospect of "abundant" products of science.
You fail to notice, livestock lives in abundance as well.
The "publish or perish" culture is going through what is essentially the same transformation brought by the invention of the assembly line. Academia, "as an industry", is experiencing what the automobile industry went through a century ago.
Grant approvals are public. Just go through them, and you will see a pattern emerge.
The argument I keep hearing is, "AI is good in the hands of a good researcher."
5–7 papers based on the same dataset? 5–7 papers on the same core topic, published within a six-month period?
They tell me that this is their core specialization and that I simply don't understand how science works.
Science is a self-governed, self-serving, and artificially defined concept.
Science is an ideal, the pursuit for knowledge for its own sake. Science is a boon for humanity because it provides an unparalleled degree of prediction and control of the natural world.
But it doesn't tell you what to do with that.
A "capitalist" will seek personal, short-sighted gain from it, socializing repercussions, no matter how devastating. They will gladly poison society and the planet and sell it to you as "your own fault". Their life-span is limited still, so what do they care?
Technological capability exceeding human wisdom is the core issue here.
You need scientific understanding on your own side in order to know and judge the alternatives beyond what vested interests feel obliged to tell you.