Newshot

Gradient Routing Creates a Headstart

Summary

Tl;dr GRAFT showed that we can make reward hacking removable. In the setup we study, a small LLM has two trainable adapters, each consisting of an additional MLP in every layer. Training updates from detected hacks are routed only into one, called FORGET, which is removed ('forgotten') after training to suppress the behaviour. We reproduced a small-model version of GRAFT and tested what makes this isolation work, where hacking concentrates across layers, and the effect of adapter width. In order to conclude that these results are general, we’d need to run larger models in different settings. Without routing, reward hacking did not reliably concentrate in a single adapter. This suggests GRAFT helps create the concentration, rather than simply determining which adapter gets the hack. Without routing, making both adapters wider made their individual hack rates more similar. This is consistent with a smaller random headstart for either adapter arising from its initial weights. Learning spe…

This source only provides a headline — there is no body to show. Use “Read the original” below to read it at the source.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗