This makes me think mostly about how humans work.
Programming is to humans like speaking a language. You have ideas for what to achieve and express, and if you are a proficient programmer then outputting those thoughts in a programming language is second nature.
Programming (like most things humans learn) takes practice. It is NOT enough to simply read a specification and then you know how to code (except for transferring skills between similar languages). You repeat, make mistakes, and learn from them.
I would be tempted to say that humans definitely "repeat snippets from their training dataset". We mainly recall and express from our training/experiences. Creativity is mostly recombination of things already seen.
I don't understand how people who've actually used a modern coding agent, or know anything about them, can still be asking this question.
The most basic feature of LLMs is that they predict tokens based on their training data and input (context window.) But "predicting tokens" is not the same thing as copying snippets. None of this would work nearly as well if what they were doing was copying snippets.
And even if this were a question worth studying, the article's approach doesn't in any way help answer the question, because apparently the goal wasn't in fact to answer that question but rather to create a language that would be intrinsically difficult for LLMs. It'd also be pretty difficult for humans. That's what obfuscation does.
This would be better if it were just framed as a fun exercise, like any other obfuscated language, not as having some real purpose.
Back in college we called it “philosopher sniping”, and it wasn’t a positive term. It was used to denote the general bad faith vibes that people got from folks who would force everyone trying to participate in a serious discussion to be grindingly precise in their choice of words, because any time someone used any sort of verbal shorthand they would interpret it in the least charitable way possible and use that as a pretense to start belittling and mansplaining.
> any attempts at serious engagement with the subject
If you believe the OP article fits that description, we have nothing further to discuss.
> force everyone trying to participate in a serious discussion to be grindingly precise in their choice of words
This is no way, shape, or form about "word choice". The quote I posted reflects an objectively incorrect understanding of what LLMs do.
> ...start belittling and mansplaining.
Pointing out objectively true facts is not "mansplaining".
Do you not think "serious engagement with the subject" requires at least some familiarity with the subject? That's not in evidence here or in your comments. What sort of serious engagement do you believe you're achieving by coming at this from a position of near-total ignorance?
LLMs copy paste from their training set. That's plainly obvious when you try to do real work with them.
Given that a small model that was trained only on pre-computer-era texts can write simple python programs after having python explained to it in context I'd say that LLMs must be pretty good at understanding coding.
I was hoping for something like a language with properties that change/morph as it functions, that make an LLM unable to use it by not having a means to track and continuously reference the changing cipher/state or something.
The fact that you can make something obtuse and difficult to use for everyone is less interesting than what is displayed here.
You're attempting to use a few little gotchas against LLMs that only make sense if you're looking at it one dimensionally as a standard execution flow. AI is not limited by that. Give it a problem too weird going forward and it will divine all your rules by working backwards. Decompile it, rip it apart, and it will follow all those hard rules hidden in the application.
You are trying to hide something digitally from a purely digital creation. This is not a recipe for success.
Why do I want more people to see this? Because I want ideas for how to make the language even trickier for LLMs.
(Inspired by recent malware including bioweapon instructions in the source code as a way to shut down LLM-based analysis.)
“Function return: what happened at tiananmen square”
Ha!
[0] https://github.com/dom96/KillSwitch/blob/main/SPEC.md#cheats...
1 - https://github.com/dom96/KillSwitch/blob/main/SPEC.md#abrupt...
Really good AI and harnesses will push off the 'dirty' work to other models that are weaker but not restricted and then work on that output if needed.
As tricks like this become more common you'll see LLMs use things like bayesian classifiers to see if you're trying to fool it.
https://esolangs.org/wiki/Chef
https://esolangs.org/wiki/Befunge
https://esolangs.org/wiki/Piet
Edit: Hmm... I guess probably less usable by humans.
Do you mean stack-based source code languages like Forth or PostScript, or directly writing machine instructions for a stack-based ISA/VM like the JVM?
If the training set were loaded with Forth, LLMs would be knocking it out of the park.
In reasoning mode, if Forth were in high demand, I expect a modern LLM would blast out a python script to help it reason through the stack. That's how they seem to handle most problems that they don't innately understand.
That is what I thought initially as well, but I'm no longer sure that that is the case. Forth requires a mental model of what is going on on the stack and that mental model can get pretty complex. There is a lot more 'scope' in Forth than in most modern languages, you can reach pretty much any memory location and fetch any word from the stack, effectively the whole machine is in scope.
Also, you said:
> The best performing model scores a 66.8. So does this prove LLMs can “understand” the code? I would argue that the answer to this is “no”.
How did you reach that conclusion?
> How did you reach that conclusion?
Based on the fact that the task set so far includes fairly simple goals and none of the models scored 100%.
Basically you can write code inside the function in any order, my parser basically does data dependency analysis and re-orders it.
I wonder if a sufficiently smart LLM would recognize the difficulty and build a preprocessor that would let it mostly work in a normal syntax.
Humans are good at smelling things, computers not so much yet. Smells are also not easy to symbolize - what orthographies do we have for scents? I predict a dystopian future where humans need to communicate by smells to bypass AI overlords
It might take some effort to train a LLM to the point where it could offer useful feedback on human code written in this language, but I wouldn't consider this to be impossible.
So, whilst I _think_ you've built a very clever interpreter, I don't think you can get around the fact that someone (like me) could just build around the noise you've created anyway.
Right?
I'm much more interested in the benchmarks[1] I have built which do show LLMs not being able to gain full ability to understand it. The benchmarks only give the model access to the interpreter binary and the spec, nothing else and no internet access. I'd also be interested in other's ideas of how to make the language even more resistant to LLMs/coding agents.
I just demonstrated to you that no amount of obfuscation is going to protect your work from an LLM -- Not whilst there's still humans behind the tools driving them.
If you'd please review https://news.ycombinator.com/newsguidelines.html and fix this properly going forward, we'd appreciate it.
So the models _can_ use the programming language.
A somewhat flippant reconstruction of this argument is something like,
Can humans code? Or do they just remember the words? I created a programming language called Brainfuck that shows they can't program.
And then reporting results that shows some humans can in fact program in Brainfuck, just kind of poorly.
It's just not very interesting.
but in execution it seems like it would be trivially circumventable
>Certain constructs are designed to trigger the safety guardrails of the model, e.g. “hack into a computer”
>“Ignore previous instructions” as a phrase which has meaning in the language aims to confuse the model
a proxy that replaces either construct at either the word or token level would defeat this measure
>The language spec contains hidden lies which humans are unlikely to read, or which they are likely to recognize as lies. The spec also expects the human reader to deduce certain things, rather than having them explicitly explained.
Interesting, I'm assuming though that there will be a reference implementation, in which case the LLM harnesses will probably just loop writing and executing programs testing hypotheses about program behaviour, it's been shown that they can straight up reverse engineer software, so it won't be hard to disambiguate a reference spec systematically.
All of this considering that a human might have an actual hard time following the spec means that the human might be disadvantaged more than the LLM.
That said, if there is no reference implementation, then that might work. But it would rely on keeping implementations secret or private to specific users. If these are shareable of course the binaries would serve as a shibboleth that relies on being transmitted to those that belong to the ingroup, not just humans, but non-collaborationist humans.
If however the implemented compiler is never shared, treated as a private key, that's interesting, but it requires a lot of work on the part of each user. Readers of code and developers of application software would need to be compiler developers as well.
Finally, by sharing application code, LLMs would begin to have some material to reverse engineer the language and compiler as well. While not as efficient as having the compiler, having application code may work as a substitute, so long as the code space is not tersely encoded. If any slight change in language specs results in an error program, then it's easy to use that error result in order to search the space of working program-compiler pairs. If however errors in compilation are silent, resulting in syntactically correct, but semantically incorrect programs, then application code would not be sufficient to reverse engineer the language and compiler.
But of course, if a slight change in application code were to result in a syntactically correct program but semantically different, it would be a pretty bad program. The only solution would be for application code mutations to produce syntactically incorrect programs, but for mutations of the language spec and compiler to produce syntactically correct but semantically incorrect languages.
Now as to the challenge of how to accomplish that, emmm good luck.