He is still an AI skeptic in as many ways that he can reasonably be, but doesn't deny the raw ability. Actually he thinks it will eclipse humans and he is a doomer.
He sees it as being an extremely empty type of intelligence though.
But I hope that people who have an intuitive understanding of contemporary machine learning (not me) will sometimes watch videos like this and think about things at a higher level. LLMs have a LOT of assumptions built in.
I think the Hofstader's view of modern LLMs is actually a deeply human and touching one. Looking at his work over the years, his curiosity has always veered towards human thought. He could have written GEB with a focus on completely different examples of self-reference, but he chose three striking humans from history. When he's describing modern systems as "empty intelligence", I think there's a little bit of heartbreak in his perspective, because he sees them as fundamentally different from humans in a way that leaves the part he loves--the "I" in the loop--out of the equation. He gave an interview a few years ago where he explains his feeling as being "diminished" not in a "What will I do if I'm not the best at math?" kind of way, but more specifically as he puts it, that humans are "imperfect, flawed structures".
The drive toward the formation of metaphors is the fundamental human drive, which one cannot for a single instant dispense with in thought, for one would thereby dispense with man himself.
It's akin to the amino acid interactions in proteins that hold biological matter together, and determine it's shape and active form. Protein folding and narrative/storytelling have strange homology :)
(I work in this area via collective intelligence, and these ideas are very dear to me during the past decade. It's neat to see the intuitions seemingly becoming validated in language models)
Hmmmmmm. While I get what you mean, and I don't disagree, please be cautious when making analogies between biological systems and more distant fields.
Yes, folding is driven by hydrophobic collapse due to interactions between residue sidechains. Really, though, we are just describing two 'complex systems', where large numbers of diverse interactions between elements leads to diverse and emergent structures.
The fact of a hole in an analogy is not in itself information. Of course there's a hole.
> I get unreasonably upset when people nitpick holes in good analogies…
What you’re describing is exactly why I strongly recommend avoiding analogy when communicating: Don’t tell me what a thing is like, tell me what the thing is. This forces really thinking about how to describe the thing precisely and succinctly. (Edit: The listener will make their own analogy.)
Very much agree. Every abstraction loses something and so is wrong in some sense, but some shave off in clever places, and allow new shortcut paths of thought and insight, sometimes to an existing place... but sometimes to a new place that was previously unreachable, or not sufficiently reachable by the necessary type of attention/minds.
> The mechanisms and math behind protein folding and an analogy between that and storytelling doesn't necessarily give us any mechanism for or deep understanding into storytelling.
Respectfully, I actually disagree (as someone trained in biochemistry).
For one example:
proteins = 1D chemical structures, peptides popped onto end one-at-a-time, that in the right chemical environment, fold into a low-energy 3D state that transforms its environment through work
stories = 1D semantic structures, words popped onto end one-at-a-time, that in the right cultural/cognitive environment, fold into a low-energy HD mental model that transforms its environment through work
This has led me to render embeddings of sliding windows of narrative streams (to investigate shapes of broadcast stories), and look for analogs to "active sites" in the way that a story interacts with the cultural medium, measured through dimensional reduction of aggregate valence reactions amongst a viewing audience.
Obviously there's more to the metaphors I use than that, but that's the one I'm leaning on to inform my work with sensemaking and map-making from valence response data. Aggregating crowd-sourced valence reactions to linear narratives gains a lot of insight by reflecting it off how we process and think about linear peptides in protein folding.
I've been developing tools inspired by these metaphors, and my research and prototypes are directly informed by what knowledge of the biological domain inspires me to port into social domain. I'm lucky to have interactions with interested parties associated with federal government, practitioners of democratic reform, and academics investigating platonic space hypotheses. I'm definitely on the edge of an ecosystem, but anything I've done of value has been guided by the metaphors that direct my attention to potential applications and research. Someone could get to my conclusions another way, but the metaphors are my cheat codes to arrive there early :)
And it goes both ways along a good metaphor -- People who develop tools in the biological sciences have told me that my porting their methods into my own distant domain has inspired them to improve their tools to ask better questions of their own biological data. So it's very fruitful work that I wholly credit to travelling both ways across an insightful metaphor. https://i.imgur.com/AeYxx7p.png
narrative -> trope + theme -> memes
I'd be very surprised to meet someone who learned that lesson by comparing protein folding (suggesting a high level of intelligence and probably education in order to understand) to storytelling, rather than having learned that stories are made up of parts in grade school or as a child just hearing, reading, thinking about, and making stories.
He described a piece of AI research (in the 60s?) where they set up a rule system - like Prolog or similar - with the mapping 'sun <-> nucleus' and 'planet <-> electron'. Then they ran the rule engine and lo! produced an analogy between the solar system and atoms.
Now this is a shallow analogy, as there is only a very weak correspondence between atoms and planetary systems. Electrons do not actually orbit in the same plane, but in more complex 'orbitals', and they are better described as probability clouds anyway.
So while yes there is a _homology_ (I re-read the parent and realised they used homology instead of analogy) between these two systems, it does not tell us much. Is there some mapping between how water molecules scaffold the folding process and storytelling? Does narrative structure tell us anything about local minima in the folding surface? I doubt it.
I do not want to be too harsh here - reasoning through analogy is fun and can be useful, it is just limited in what it can do, especially the further apart the systems are.
https://george-lakoff.com/books/metaphors-we-live-by/
Took a course with Lakoff as an undergrad and it was compelling.
(The post primarily emphasizes self-referentiality rather than analogy, but I suspect similar things could be said about analogy.)
I don't see why Conway's game of life should not be considered self-referential though... I mean it's isn't it Turing complete? I don't see how any definition of self-referentiality should require throwing out systems which are minimally turing complete... If Turing complete is not enough, doesn't that imply that computable artificial intelligence is impossible in the first place?
All this being an empirically unproven theory/hypothesis. But an extremely strong one (if you ask me).
> The big, old ideas about intelligence that ended up basically vindicated were the ideas about how intelligence is about prediction, and prediction is about compression, and compression is about finding better and better upper bounds on Kolmogorov complexity.
Well, sure - prediction works as a baseline, if you abstract out everything else about what counts as intelligence and subsume it under this framework. Any protocol for intelligence can be entirely reduced down to this without trying to understand anything about the structure of intelligence. Solomonoff induction is "vacuous" in this sense too - it doesn't try to understand any intension about the turing machines it finds simple, it just brute forces over all of them. So you're making a claim about intension, whether you want to or not.
It's really no different than say, Darwinians, saying, "what matters is victory at the end". I think the statement has value in the context of some discussions; if the discussion has veered too far in one direction, push as a reminder. That's good. But trying to make grand universal statements like this makes it vacuous.
It's one of these classic unfalsifiable too general statements. The way the end of the article is framed is also icky - the way I'm reading it, it needs to prove all the old curmudgeon evil theories wrong. It reminds me of internet debates around what "the scientific method" is or whatever. It has this kind of zeal that needs to pit itself against the "enemy" and assert itself as the sole right viewpoint, even when say, naive "let's just be empirical" is wrong (e.g. recently had a discussion here about Mach and Boltzmann about this and had a similar interaction).
---
Basically, "shut up and calculate" type theories are never correct, and furthermore, you yourself don't shut up and calculate, or think that way at all, and a higher level intelligence won't conceive itself as operating on that anyways. So what are we doing here? It seems like a way to get mad and feel like an intellectual victim.
I’m confused why there seems to be a dismissal of the most basic ‘strange loop’ of the LLM - the fact that it’s evaluating a context to choose the next word, then reevaluating in a context where that word has been appended.
That always seemed to me like the essence of a Hofstadterish strange loop, so the emergence of Hofstadterish phenomena (self rep, etc) doesn’t seem surprising.
And if anyone's reading this and hasn't read Hofstadter, you're making a mistake, it's utterly perspective-changing stuff. Well, it was for me, at least.
Then, after I graduated from college, a friend of mine who was finishing his physics degree mentioned that he had just finished it and that it was life-changing. I read it again and deeply regretted having put it down for those 12 intervening years.
At first, I flipped through the book looking at the Escher prints. My second time through, I read the dialogues. Next I had a go at reading the chapters and got lost. Maybe on my fifth read-through (years later) I actually attained an understanding of Gödel's Theorem (at least for a brief moment before it vanished again).
I credit GEB (and its dialogues like Little Harmonic Labyrinth) for helping prepare me to deal with the kind of multi-level abstraction involved in developing a language interpreter or machine simulator. Especially that time I had to debug a crash that only occurred when running my simulator inside a simulated instance of itself (i.e. when three levels deep, but not two).
Weirdly, Aaronson doesn't even seem to be conflating the two:
> Consciousness and subjective experience of course remain extremely mysterious.
I think he's just misreading Hofstadter as stating that cognition depends on self-reference?
First, LLM AI systems have incredibly huge blind spots despite their incredible performance on many tasks, so self-reference might be the key to what's missing (or not). For example, an LLM AI just solved Navier Stokes, but could not explain the LEAN proof, while a human could.
Second, Hofstadter had more than one idea about intelligence and the mind (see the OP topic of this HN discussion!), and LLMs are quite on-point regarding analogy-forming.
So it may be well be that self-reference and analogy are both part of intelligence, and self-reference is missing and that is leading to major weaknesses.
Third, Aaronson links to a (paywalled) Hofstadter essay form 2023, which was eons ago in AI, and from the intro it seems to be about the sadness of AI replacing humans, not a disparagement of AI ability.
"an LLM AI just solved Navier Stokes"
I assume you mean:
"an LLM AI [company] just [claimed that a team of mathematicians they hired, using their AI] [may have] solved [part of] Navier Stokes[, definitely prompted by (and possibly by looking at) the work of human mathematicians."
> But the idea that you’d need explicit self-referentiality before you could get convincing and world-changing conversational intelligence? Let it be buried in a Westminster Abbey or Arlington National Cemetery for the most important wrong ideas in human history
I am not as confident as you that an LLM cannot explain the lean proof of Navier-Stokes. Rather, I would expect human mathematicians to try and understand the proof without assistance, so as to obtain community understanding in a lossless way.
Hofstadter generally seems depressed about the possibility that human cognition is not so special or complicated, and that AI/LLMs may have replicated or even surpassed it. Here’s another piece from 2023 of his: https://www.lesswrong.com/posts/kAmgdEjq2eYQkB5PP/douglas-ho...
Q: How have LLMs, large language models, impacted your view of how human thought and creativity works? D H: Of course, it reinforces the idea that human creativity and so forth come from the brain's hardware. There is nothing else than the brain's hardware, which is neural nets. But one thing that has completely surprised me is that these LLMs and other systems like them are all feed-forward. It's like the firing of the neurons is going only in one direction. And I would never have thought that deep thinking could come out of a network that only goes in one direction, out of firing neurons in only one direction. And that doesn't make sense to me, but that just shows that I'm naive.
It also makes me feel that maybe the human mind is not so mysterious and complex and impenetrably complex as I imagined it was when I was writing Gödel, Escher, Bach and writing I Am a Strange Loop. I felt at those times, quite a number of years ago, that as I say, we were very far away from reaching anything computational that could possibly rival us. It was getting more fluid, but I didn't think it was going to happen, you know, within a very short time.
And so it makes me feel diminished. It makes me feel, in some sense, like a very imperfect, flawed structure compared with these computational systems that have, you know, a million times or a billion times more knowledge than I have and are a billion times faster. It makes me feel extremely inferior. And I don't want to say deserving of being eclipsed, but it almost feels that way, as if we, all we humans, unbeknownst to us, are soon going to be eclipsed, and rightly so, because we're so imperfect and so fallible. We forget things all the time, we confuse things all the time, we contradict ourselves all the time. You know, it may very well be that that just shows how limited we are.
A good place to start for anyone that is interested is his book Metaphor's We Live By.
I will never not find it amusing how a simple thing like money is obscured or "spun" by a thin veil of pseudointellectual bullshit.
And it's relevant today given how people like to anthropomorphize LLMs, and compare biology to digital machines we make. Of course there are similarities. But the point about metaphorical thinking is we are mislead by treating metaphors as literally true.
Anyway if the stronger claims in the book are at all true, it might impact how an alien species thinks differently than us (given a lot of our metaphors are biologically based). And could present significant difficulties for decoding an alien signal.
Whether it is the evolved shape of a flying creature or an abstract depiction of a meteorite (or even a comet!), there is a kind of intuition about the sharp end and the flaring of air or fluid around it, with some sort of trailing tail from whence it came...
One can imagine many observers who could find our space probes would also have some of these experiences?
Of course, all of this assumes some kind of visual perception and cognition to even recognize the plaque as having markings made with an intent to communicate some abstract ideas...
But I think that LLMs are bad at something that has to do with taking seemingly disparate concepts and assimilating one into the other to convey a novel idea.
Like these articles...
<https://spectrum.ieee.org/jimi-hendrix-systems-engineer>
<https://zed.dev/blog/agentic-xanadu>
...are anything but convincing once you read past the gravitas that the LLM lends the prose.
Metaphors are at the core of LLM cognition too via vector embeddings. The distance between embeddings is an inverse measure of their metaphorical attachment. That lets AIs scrape meaning from natural language, which demonstrates that natural language is an encoded sequence of metaphors.
Anyway, ML was already onto analogies with word2vec, which famously could answer questions like "man is to woman as king is to ____" mathematically. This stuff seems quaint now.
People with more knowledge, especially practical working knowledge over many fields, tend to have much more freedom in finding solutions.
Now, an interesting question is how good at LLMs are at analogy, especially deeper transferable concepts?
Edit: it actually is very possible GP wasn't being disparaging, in which case truly sorry! I'm not trying to be too snarky.
From what I recall, to Hofstafer analogy making isn't some higher level cognitive process, certainly not a language based one, but basically is THE cognitive process all the way from perception on up, and is the mechanism by which we form object categories in the first place.
As always Hofstader's ideas are interesting, but I can't say I agree with them. It seems that the key evolutionary benefit, and function, of a brain is prediction, which is the superpower that moves us from being stuck in the present to being able to "see" (predict) the future, and therefore from being merely reactive to being able to proactively plan and predict future outcomes (what will the sabre-tooth do, where is the water supply?) based on our experience.
Given the never-same-twice nature of sensory perception, before you can predict you need to be able to generalize/categorize, which I think Hofstader would regarded as analogy making (how is this thing I'm seeing similar to what I've previously seen?), although it seems the actual mechanism involved is embeddings or embedding-like representations where similar inputs have similar representations, and what might more simply be considered as associative recall provides the generalization from view/instance to identity/category.
So, is it really analogies all the way up, or are our perception and cognitive processes better regarded as generalization and prediction, which seem not only seem to have direct and obvious neural realizations, but also match the evolutionary needs that we would expect to exist?
This is so funny to me, because as many people know, sharing an analogy with another person is the fastest way to LOSE an argument with someone, or otherwise spiral it into an unproductive place.
I think it’s Scott Adams who used to say analogies work well for explaining. They work terribly for persuasion.
I just experienced this in a conversation. An analogy offers an opportunity to engage with the straw man and miss the forest for the trees.
Ex:
Explainer: "Getting a spleen means cutting open the patient and taking it out. It's just like how I'm going to unzip this section of the patient-shaped doll, and remove this little purple bean. In both cases a hole is necessary in a similar location."
Hostile listener: "Nonsense! I can just buy beans at the store! So just buy a spleen! No hole!"
I’m not sure if Hofstadter puts it this way, but to me even the core aspects of your sentences in this post have roots in analogies. What does it mean to lose an argument or to spiral it to a different place? There is no place, there is no lost item, but we talk about these abstract ideas in ways that largely depend upon understanding things like physical objects and space and movement.
If he had accepted the "Bitter Lesson", I think he would have been at the forefront of LLMs.
The path forward all big llm providers ("ai" labs) have gone is neuro-symbolic (even though they publicly would never labeled it as such to not admit critics like Gary Marcus were right - even though all their actions actually point in that direction).
Smolensky's latest paper posted here the other day has some thoughts on how modern neural networks might beconsidered neurosymbolic, or rather "gradient symbolic processing," from another perspective entirely.
I wouldn't say the bitter lesson has given out! If you haven't noticed, these things keep getting bigger and bigger.
If the workings of those circuits are obvious to you, I'd really like to learn. Do you mean the level of analysis at https://transformer-circuits.pub/ ? (That looks like good work but not a deep understanding.)
Hofstadter referenced this back in the day as a promising beginning: https://en.wikipedia.org/wiki/Sparse_distributed_memory which sounds kind of similar in style to the embeddings you bring up.