Essentially it goes "You know how your model can remember its training data? Well, what if you treated its recent context like more training data and updated (some of) the weights using (mostly) the same process used to train it?"
The end result is very good at remembering things but also really good at adapting to new unseen distributions.
This would create a three-layer memory system:
- Stable long-term memory (initial base weights)
- Mid-term memory built from the compactions and replay buffers
- Short-term memory (KV cache)
Sleeping would just be a fancy term for consolidating and transferring information from one memory layer to another during offline hours. Maybe that's also what the brain does while sleeping.
Scaling test-time compute has emerged as a key ingredient for enabling large language models (LLMs) to solve difficult problems, but comes with high latency and inference cost. We introduce sleep-time compute, which allows models to "think" offline about contexts before queries are presented: by anticipating what queries users might ask and pre-computing useful quantities, we can significantly reduce the compute requirements at test-time. To demonstrate the efficacy of our method, we create modified versions of two reasoning tasks - Stateful GSM-Symbolic and Stateful AIME. We find that sleep-time compute can reduce the amount of test-time compute needed to achieve the same accuracy by ~ 5x on Stateful GSM-Symbolic and Stateful AIME and that by scaling sleep-time compute we can further increase accuracy by up to 13% on Stateful GSM-Symbolic and 18% on Stateful AIME. Furthermore, we introduce Multi-Query GSM-Symbolic, which extends GSM-Symbolic by including multiple related queries per context. By amortizing sleep-time compute across related queries about the same context using Multi-Query GSM-Symbolic, we can decrease the average cost per query by 2.5x. We then conduct additional analysis to understand when sleep-time compute is most effective, finding the predictability of the user query to be well correlated with the efficacy of sleep-time compute. Finally, we conduct a case-study of applying sleep-time compute to a realistic agentic SWE task.
We're talking about a step that updates weights based on say between 10k and 1M tokens.
Biologically humans do similar compression, so introducing a similar concept to an LLM also feels reasonable. Hardware isn't fast/cheap enough to do this on an ongoing basis, similar to how it's too expensive for our brains to do this while we're moving through the world.
All we have now most of the time in LLMs is "working memory" we're missing a lot of the functionality that allows for episodic memory and selective plasticity.
The more you read about how human brains work, the more you realize that we may have figured out a piece with LLMs, but it's certainly nothing approaching AGI. People insisting so are blowing smoke for investor hype or don't understand a big piece of the concepts involved.
Does a motor vehicle get "sleep" when it is serviced? When I reboot a computer, is that equivalent to a nap?
> In animals, the transfer from short-term memory to long-term memory is thought to be supported by hippocampal replay [33], especially during sleep [41]; in this phase, short-term hippocampal memories are reactivated and consolidated into cortical synaptic weights. Sleep makes animals unable to respond to external stimuli, suggesting that it must provide enough cognitive benefit to justify this cost [41]. Inspired by these biological processes, we propose a method for transferring context-window memory into persistent weights. When the model’s context window becomes full during inference, the model enters a “sleep” in which it performs multiple forward passes over the accumulated context and recursively updates its fast weights via a learned local rule. As in animal sleep, the model receives no external input tokens during this phase. After consolidation, the context window is cleared, and the model resumes operation with updated fast weights. During training, the model is optimized end-to-end by backpropagating through the entire process to maximize task performance after sleep.
One thing we do know for certain is that it is necessary, it is needed in "dumb" animals as well as in you and I. If an animal can't sleep it will eventually die.
I don't think that applies to the activity described in the OP. Does their LLM "die" if it can't perform the function described?
That turns out to be un-settled science. No human has ever died from lack of sleep.
People point to “fatal familial insomnia” as a counterexample. But they die to the disease, not the lack of sleep.
In a series of controlled experiments, rats and fruit flies did die from lack of sleep. But no one has yet proven that it holds true for vertebrates except for rats.
In other words, it could be true that “among vertebrates, only rats die of sleep deprivation.”
So “if an animal can’t sleep, it will eventually die” is actually quite hard to prove, and depending on how you look at it, somewhat easy to disprove by the fact that rats and fruit flies were so difficult to kill from sleep depravation alone.
Personally I’m skeptical of the rat study too. Claude amends this:
> What they did not establish: the mechanism. On autopsy, “no anatomical cause of death was identified.” The rats showed weight loss despite eating more, body temperature problems, and skin lesions, but nothing that pointed to a clean cause. So no, they could not say a rat “died from sleep deprivation alone” in the sense of identifying what sleep loss did to the body to kill it. They showed a strong association under tight controls, not a proven causal pathway.
As far as I understand it, there is a disease that destroys your brain's ability to produce sleep. Once you have it, you suffer total, progressive insomnia and die within roughly 6–18 months. Scientists debate whether it's the underlying brain damage or the sleeplessness itself that causes death, but the two are inseparable in practice, and sleep deprivation is considered the leading candidate.
Separately, the longest anyone has stayed awake under controlled conditions was 11 days, which produced severe cognitive impairment, paranoia, and hallucinations; suggesting the body deteriorates rapidly without sleep.
It's probably not wise to state your original claim as established fact.
> People point to “fatal familial insomnia” as a counterexample. But they die to the disease, not the lack of sleep.
It’s a prion disease. It’s established fact that they don’t die from the lack of sleep.
It’s not a small quibble to point out that the central argument (“animals need sleep or they’ll die”) may be mistaken.
Very few animals fail to eventually die even with as much sleep as they want.
But before death, there is a loss of cognitive function from sleep deprivation, and we observe this too with AI whose context windows get too full.
While we don't know very much about sleep, my understanding is that we do have a long list of things that we do during it, we just don't really understand if sleep is necessary for each of them or simply a convenient opportunity for it.
There's lots of things biology does in response to easy-to-detect proxy signals instead of the real thing they care about: Our sensation of needing to breathe more is based on too much carbonic acid in our blood, not lack of oxygen, which is why in general nobody is allowed in an elevator with a liquid nitrogen dewar; Our natural distaste for incest is based on who we grew up with, not our actual DNA; Get too cold and some people suddenly feel warm and want to (and some do) take all their clothes off even though that would just make them hypothermic even faster.
Being asleep may trigger the things we need to get done, but that doesn't mean sleep is *fundamentally* necessary for the things we need to get done. It could be just that it happens to be the way our biochemistry is wired, and we may find some other way to trigger those things.
The quotation given by djeastm would by my guess for what a dream is, and why we have them. But we don't spend all our time asleep, dreaming. And I'd be the first to say that my guess isn't worth much, as I'm not a brain scientist.
Meaning: it might just provide a big advantage.
I don't want to overextend and assume that any advantage extends to LLMs. That rest-and-recuperate advantage might also extend to LLM-based AIs. Or maybe not, and the rest-and-recuperate is mainly useful for biology-based organisms. But there is some logic to it.
> The function of sleep in animals is largely obscure.
In my understanding, it's well-understood that sleep is used to consolidate and store long-term memories (amongst other functions, like cell and muscle repair). They've found this memory-consolidation-during-sleep even in relatively simple animals like bees.
You're talking about different things: biological necessity and evolutionary benefit.
You can find out about the former by preventing an animal from sleeping (but otherwise provide all other needed things), and seeing if it will eventually die.
That is actually almost impossible to do. The rat study was as close as we’ve ever come, and it’s still debated whether the rats died due to lack of sleep or some other mechanism, since the autopsy couldn’t confirm a cause of death. (It could have been due to the way the experiment ran, for example, not the lack of sleep.)
If you don't periodically clean the context, an LLM effectively goes insane in terms of outputs.
If the LLM were fully controlling a physical system (like a robot body) that contained it the resulting insanity of an ever-growing, never cleaned context would likely result in some sort of death-like event.
Also, there's different kinds/stages of sleep, which probably perform different functions.
For instance, REM may do something like the GP describes, consolidating memories and processing learning. Deep sleep may do something else (I vaguely recall some stage of sleep is used by neurons to clear certain waste products).
i feel like its confusing to reuse the word for a process that aims to deliberately change state of the machine / process
That's more like a doctor visit and a workout. The sleep will be the part of the duty cycle when it's not operating.
> When I reboot a computer, is that equivalent to a nap?
Yes, it wakes up completely refreshed and in good working order, usually, and if there's still a problem you know you need a technician.
Here the analogy isn't without reason.
Clicking through, that’s exactly what it is. Seems like “sleep” is an excellent term to use here.
I mean, you do put your computer into "sleep" mode and then "wake" it.
Analogies are useful. I think we need to learn how to continue to benefit from them despite the risk of anthropomorphication.
it is very non-helpful—or worse—to use this language, this way.
One might as well say "need neural plasticity" which is as much an analogy and equally misleading and counterproductive in shaping the right model of the system.
One might even call this pernicious, what it encourages is already a social problem; and it doesn't aid understanding, it confounds it.
Maybe someday we'll understand the way our minds work well enough to design from first principles but until then we've only got one template for how a thinking machine should look.
At the very least, we know that sleep and dreaming do exist in biological brains. (Doesn't mean any of it is applicable to artificial neural nets, doesn't mean it'll work for our specific architectures etc. etc., but at least the idea requires fewer assumptions than a completely untested novel theory.)
There is a strong, non-trivial connection here between what your brain does in sleep and what they are studying.
You wouldn't object to referring to robot eyes or robot legs.