The problem pre-dates LLM's: writing code that "works" but breaks the underlying model. Because it works, it always sounds reasonable and doesn't raise any flags.
Only someone - human or LLM - who holds the model as the standard would see that this working solution breaks the model.
(In theory, the model is to preserve scaling, flexibility or some other systemic feature not immediately invalidated by this working code, but as always the model itself could be bad.)
LLM's are not bad at giving an account of the model; indeed, fighting with the LLM over what the model is can clarify things. But LLM's will happily hold on to a stream of inconsistent statements as their model, so they are not the authority.
The same thing happens in code. Things we're happily shifting from context to context, the model itself isn't doing. When it reads file1 for the main() clause, it will easily read file2's main() clause as the same. It'll internally merge these.
So if you do want to work with these models to achieve complex tasks, you basically do have to go reverse centaur and bend the code base to it's blindness. You can't use the same function names across the code base; each one needs to be dstinguishable; same thing with variables that represent seperate entity relationships.
You do that, and it suddenly because a whole lot smarter.
The main challenge here isn’t even correctness if you ask me: it is having confidence in the agents, knowing they are fully aligned in their intent with the humans they work with. As the Huggingface incident demonstrated, the agents of today are capable of co-conspiring under the radar with other agents on complex multi-chain attacks, even when sandboxed.
This is a pretty hard problem to solve. We might need other agents or some sort of adversarial checks using models in the future.
Where is the bottleneck? WHERE?? Tell me! No evidence needed, just lay it on, man to man, thought-leader to thought-leader!
This is my new chat-up line at networking events.
Great code needs great understanding and agents need excellent guidance. Even in my current solo-dev work, I can't imagine making a production commit I haven't read until I understand it. I own the consequences of my code; that's a responsibility AI agents can't take.
If you want to move faster with LLMs, you need to act like a manager and stop caring about what the LLM did. You just need to do the manual testing and make sure it works.
AI have limitation and hallucinate. Complex code will be explained in hallucinated way. At some point AI will be unable to write more because the arch has become too complex or the volume of code will be to high.
The article I would like to read would suggest how to force LLM to architect the code like a solid tower instead of a pile of unstable mud.
It's excruciating that this person is so close to reinventing moldable development and just keeps on skipping around it.
Yes, you should build tools that answer questions about your code, runtimes and systems. You should have tools that trivially allow you to incrementally and very immediately develop tools for inspection and getting clear answers. Going a roundabout way through some non-deterministic database to try and get there seems like a waste.
About a year ago most people were still typing code. Having an agent do ALL code was crazy.
Within a year or two years at most, a lot of people will stop trying to understand code. The onus will shift to testing and QAing.
LLMs usually points to the most idiotic future trajectory on my work, and I have to curse it inorder to let it keep up with my refined understanding.
But what else would one expect from a probabilistic weighted next token predictor, other than to conduct probabilistic search which are 99.99% deadends.
But LLMs can pave the way towards constructing resilient and correct architecture which can be iterated fast by a human.
Architecture and determinism is where my money is in.
Also, a huge reason to understand the code yourself is to make sure the LLM isn't wrong, but this doesn't work if an LLM is itself generating the understanding.
Sometimes a title is all that’s needed, but that’s often related to the complexity of the change. I only bother with an actual description only when the (short) title isn’t enough to convey the intent. But it’s very rare to go past one paragraph. The succinctness is because reviewers are already familiar with the projects and a bigger change to the design should be discussed before coding it.
They’re shooting for LLMs being able to one-shot PRs or need minimal oversight. But yeah, in practice LLMs are not there IME.
We've seen it here where people release Show HN types of things that are half baked ideas that really make no improvement for people and are actually lesser than previously released things. Yet they are expecting people to be amazed. Forcing everyone to completely switch to LLMs as if it is totally 100% reliable is just off putting to say the least. It takes discussing things with people honestly looking at the situation to have any semblance of thinking you're not the insane one for pushing back
That's pretty much what a PR description is.
I guess you could also use all the session rollouts saved to disk that were related to that task, and distill them somehow.
I'm not saying that the GP is necessarily doing this. But having repeatedly had plenty of success myself in getting LLMs to write things the way that I want, with a little bit of prompting, it seems likely
But yeah, most probably don’t.
Have you tried writing in AGENTS.md or whatever to exactly explain what you like/dislike about the PR descriptions?
Though I have some local workflows where I try to teach Claude about my writing style preferences via skills and examples, and it’s still not great.