And for that matter, what’s it do with 9 digit numbers? Like, is it more accurate with them, or are these little guys mainly good at adding numbers with exactly 10 digits?
Basically, are the failures modes a gentle increase in inaccuracy, or spectacle failure outside their parameters?
The set of 11-digit numbers with any given failure mode (or even successful output) has no discernable pattern, merely whatever randomness the training process baked into the model.
You can't predict ahead of time when they will fail spectacularly, nor draw a clear boundary around the failure cases. And early major example of this were the "glitch tokens" introduced into most LLMs by training on reddit data.
But there is an "in general"/"average failure rate across all inputs of a given size" answer: LLMs performance drops off a cliff once the input reaches too much complexity. (A "┐" shaped curve) In contrast to humans, where you can ask a child to add two N-digit numbers and the error rate will be approximately linear to N.
For instance the current high score model (311 params [0]), when given 12345678900 + 1, responds with 96913456789.
An interesting experiment would be: what's the minimum number of parameters required to handle unbounded addition (without offloading it to tool calls).
Of course memory constraints would preclude such an experiment. And so a sensible proxy would be: what kind of neural-net architecture and training would allow a model to handle numbers lengths it hasn't been trained on. I suspect, this may be not be possible.
Does this result suggest that if we had N clever humans manually building an LLM, they might come up with something as smart as a frontier model, but potentially 45 times smaller? (1644 / 36 ~= 45, N = very large, time not specified)
True, but with even smarter humans, you could exploit the interactions for additional calculations.
While it sounds a bit silly, it is one of the hypotheses behind a fast takeoff. An AI that is sufficiently smart could design a network better than a trained one and could make something much smarter than itself on the same hardware. The question then becomes if that new smarter one can do an even better job. I suspect diminishing returns, but then again I am insufficiently smart.
I think it would be interesting to see challenges where two networks are trained and evaluated on the exact same datasets and the architecture is the same except for the presence of self-attention layers in one network.
So far it seems to me that self-attention really brought new capabilities to a network - essentially change the network's functionality in response to the input. It would be interesting to see if there are problems (i.e. datasets) that a "traditional" feedforward network fails to solve, but a transformer network of the same size can solve.
My guess would be: yes there are, and they are the kinds of "variable task" datasets that we see with LLMs, i.e. where part of the input indicates the task itself and part indicates the data for the task.
Do we have a layman explanation for what makes self-attention so uniquely powerful? Something more than "it lets you do self-attention".
So (modulo a _lot_ of details) it increases the power from that of a "calculator" to that of a "computer".
I wonder why they don't just write the code themselves, so by design the focus can be on the model.
It might work, I considered running a test like this. But it does demand certain things.
The subnetwork has to be either crafted as "gradient resistant" or remain frozen. Not all discovered or handcrafted circuits would survive gradient pressure as is. Especially the kind of gradients that fly in early pre-training.
It has to be able to interface with native representations that would form in a real LLM during pre-training, which is not trivial. This should happen early enough in pre-training. Gradients must start routing through our subnetwork. We can trust "rich get richer" dynamics to take over from there, but for that, we need the full network to discover the subnetwork and start using it.
And finally, it has to start being used for what we want it to be used for. It's possible that an "addition primitive" structure would be subsumed for something else, if you put it into the training run early enough, when LLM's native circuitry is nonexistent.
Overall, for an early test, I'd spray 200 frozen copies of the same subnetwork into an LLM across different layers and watch the dynamics as it goes through pre-training. Roll extra synthetic addition problems into the pre-training data to help discovery along. Less of a principled solution and more of an engineering solution.
Similarly I’m always surprised that we don’t start by training a small set of layers, stack them and then continue.
One of the main issues is: we don't know how to generate useful computational structure for LLMs - or how to transfer existing structure neatly across architectural variations.
What you describe sounds more like a "progressive growing" approach, which isn't the same, but draws from some similar ideas.
Without any formal verification: The input space of two 10-digit numbers is a bit bigger than 64-bits, so exhaustively verifying all possible inputs doesn't sound practical. Using the same subset of the input space for verifying each submission seems like the easiest way to be fair, and not disclosing that subset to the competitors is obviously necessary.
That little "add" of yours has the overhead of: having an LLM emit it as a tool call, having to pause the LLM inference while waiting for it to resolve, then having to encode the result as a token to feed it back.
At the same time, a "transformer-native" addition circuit? Can be executed within a single forward pass at a trivial cost, generate transformer-native representations, operate both in prefill and in autoregressive generation, and more. It's cheaper.
Although the converse would be interesting, racing city buses.
[A B]
[1]
[1]
[A+B]I guess the analogy there is that a 74ls283 never really has a number either and just manipulates a series of logic levels.
99% reliable means you still can't remove the human from the loop — because you never know which 1% you're in. The only way to actually trust output is to attach a verifiable confidence
signal to each response, not just hope the aggregate accuracy holds.
We built a local gateway that wraps every LLM output with a trust envelope: decision trace, risk score, and an explicit PASS/REFINE/ESCALATE/BLOCK classification. The point isn't to make
LLMs more accurate — it's to make their uncertainty legible so the human knows when to step in.
Open source if you want to look at the architecture: github.com/base76-research-lab/operational-cognosYep, for people who don't have know the fundamentals (i.e. maths). To people who don't know the universal approximation theorem, this may seem like "magic", but it's just as much magic as making a dark room bright by flipping a light switch.
When you hand-code the weights, you're essentially implementing a known algorithm (carry-propagation) directly into the network topology. But trained networks often discover distributed representations that spread the computation across more parameters in ways that are harder to interpret but more robust to input distribution shifts.
I'd be curious whether the 311-param trained model generalizes better to bases other than 10, or to addition with different digit counts than it was trained on. In my experience, the 'messier' learned solutions sometimes capture more structural regularity than the clean engineered ones, precisely because they aren't locked into a single algorithmic strategy.
Seems the castle of cards isn't just high enough. /s
I was initially excited until i saw that, because it would reveal some sort of required local min capacity, and then further revelation that this was all vibe coded and no arXiv, makes me feel I should save my attn for another article.
Does this boil down to a condemnation of all scientific endeavours if they use resources?
Would it change things if the people who did it enjoyed themselves? Would they have spent more energy playing a first person shooter to get the same degree of enjoyment?
How do you make the calculation of the worth of a human endeavour? Perhaps the greater question is why are you making a calculation of the worth of a human endeavour.
Now if you said this proof of addition opens up some other interesting avenue of research, sure.
Well for starters, it puts the lie to the argument that a transformer can only output examples it has seen before. Performing the calculation on examples that haven't been seen demonstrates generalisation of the principles and not regurgitation.
While this misconception persists in a large number of people, counterexamples can always serve a useful purpose.
But it does not, right? You can either show it something, or modify the parameters in a way that resemble the result of showing it something.
You can claim that the model didn't see the thing, but that would mean nothing, because you are making the same effect with parameter tweaks indirectly.
Iteratively measuring loss is a way to reconstruct values. That's trivial to show for a single value If 5 gives you a loss of 2 and 9 gives you a loss of 2 then you know the missing value is 7.
A model with enough parameters can memorise the training set in a similar manner. Technically the model hasn't seen that data by direct input either, but the mechanism provides the means to determine the what the data was. In that respect it is reasonable to say the model has seen the data.
Performing well on examples not in the training set is doing something else.
Any attempt to characterise that as having been seen before negates any distinction between taking in data and reasoning about that data.
So I don't understand how any one can make the claim that the model as not seen it. Because the internal transformation is similar.
By what mechanism do you propose the model observed the test set?
By explicitly setting the model parameters.
What happens when a model is trained? We tweak the model parameters by some feed back.
In both cases, you affect the model parameters. Only the method is different. So both are eqvialent to "model observing the test set".
Are you trying to say that the person who entered the parameters had access to the test set? I find it more likely that they encoded the generalising rule than observed every instance of its use.
Those who worry about an imaginary risk and live their lives in constant fear have turned into nothing more than machines enslaved by propaganda.
I think that's one very good reason to make them more efficient, and that's part of the point of contests like this one.
https://www.reidatcheson.com/transformer/llm/ml/cuda%20graph...