Researchers Propose Sleep-Like Consolidation for Large Language Models

Researchers have introduced a sleep-like consolidation mechanism for large language models (LLMs) to address the scaling limitations of their attention mechanisms in long-horizon tasks. The proposed method involves the model periodically converting recent context into persistent fast weights before clearing its key-value cache. During this "sleep" phase, the model performs offline recurrent passes over accumulated context and updates fast weights in its state-space model (SSM) blocks. This approach shifts computation to the sleep phase, aiming to preserve the latency of wake-time predictions. The effectiveness of this mechanism was tested on synthetic tasks and a math reasoning task, with results indicating that increased sleep duration led to improved performance, especially on tasks demanding deeper reasoning.