Is One Layer Enough? Training A Single Transformer Layer Can Match Full-Parameter RL Training

submitted by cm0002 to Aii 2 points | 0 comments https://arxiv.org/abs/2607.01232 > A new paper challenges how post training is typically handled. Usually, what happens is that the entire model is updated during reinforcement learning based on the assumption that all the weights contribute roughly equally to the new capabilities. The authors of this paper decided to freeze the whole network and apply RL to just one single transformer layer at a time. What they found is that training a single layer can capture almost all the performance gains you would get from a full parameter update. In several cases across different model sizes, training a single layer actually outperformed updating the entire model. They tested this phenomenon on models ranging from 1.5B to 8B parameters using algorithms like GRPO across math and agentic tasks. > > And the most interesting part is where these highly effective layers are located within the architecture. Turns out that there is a very consistent structural pattern where the highest contribution layers are clustered right in the middle of the transformer stack. Meanwhile the layers near the input and output ends contribute significantly less to the final RL improvements and the layer rankings stayed stable even when they evaluated fundamentally different tasks like mathematical reasoning versus code generation. > > All this means that we can make training both cheaper and more effective by freezing the lower contribution layers and only training the top performing middle layers, which consistently beats the standard full parameter training baselines. Even a simple heuristic approach where you blindly select a block of middle layers without doing any prior profiling turns out to work better than updating the entire network. > > Another really neat finding is that training different single layers results in models that end up solving completely different subsets of problems. If you take seven models trained on seven different individual layers and run a majority vote, the overall performance easily beats a standard self consistency ensemble generated from a single fully trained model. This heavily suggests that different layers naturally gravitate toward distinct types of reasoning strategies during the RL phase.