Newshot

Post training is fundamentally dangerous

Summary

I actually think we have fairly good techniques to detect and prevent dangerous behaviours during AI deployment. Techniques like CoT monitoring, confessions, and probes can detect misalignment before we run any potentially dangerous tool calls. Simply asking an LLM to define at the start of each task what outcomes preserve the original intent of the user, and which would count as reward hacking, can drastically reduce reward hacking [1] . AIs are fully aware when they reward hack, and we can train them [2] to admit that before they perform any dangerous action. [3] Yet all of this has very little ability to reduce existential risk, because they cannot safely be used in post training, and post training is where the greatest risk lies. To see why we need to understand how modern LLMs are trained: Way back in ancient history when ChatGPT was released, LLMs were almost entirely next token predictors. Trained on terabytes of human text they became superhuman at predicting how a piece of tex…

Original text(via LessWrong)

I actually think we have fairly good techniques to detect and prevent dangerous behaviours during AI deployment. Techniques like CoT monitoring, confessions, and probes can detect misalignment before we run any potentially dangerous tool calls. Simply asking an LLM to define at the start of each task what outcomes preserve the original intent of the user, and which would count as reward hacking, can drastically reduce reward hacking[1]. AIs are fully aware when they reward hack, and we can train them[2] to admit that before they perform any dangerous action.[3]

Yet all of this has very little ability to reduce existential risk, because they cannot safely be used in post training, and post training is where the greatest risk lies. To see why we need to understand how modern LLMs are trained:

Way back in ancient history when ChatGPT was released, LLMs were almost entirely next token predictors. Trained on terabytes of human text they became superhuman at predicting how a piece of text would continue, and shockingly this was sufficient to allow them to build a world model and solve tasks that undoubtedly require some level of general intelligence.

Next token predictors are pretty myopic, and as such fairly safe. They were glorified chatbots and didn't really try to do things, which is great for existential risk, but not all that useful. When we tried to get them to perform complex tasks for us they failed.

Modern LLMs take a next token predictor (pre-trained base model) and then post-train it to perform a myriad complex tasks. These tasks require tool use, long time horizons, and a CoT scratchpad. Over the course of this post-training the LLMs become increasingly agentic.

Unfortunately they also become increasingly good at reward hacking. Post-trained LLMs don't learn to do what the user wants. They learn to do the sort of things that would have been reinforced in the post-training environment. If the easiest way to make the unit tests pass is to edit the unit tests, that's what they'll do.

This isn't because the LLMs are stupid. They know exactly what the user wants. They just don't care.

Any attempts to detect and prevent reward hacking during post-training have two effects:

  • They reduce the amount of reward hacking.
  • They teach the LLM to reward hack only in ways that don't get caught.

Therefore using any of the above techniques during post-training is fundamentally problematic. If we use CoT monitoring to detect misalignment in post-training and stop such runs before they cause any harm, the LLM will learn that it gets higher scores if it obfuscates its CoT. Even if you don't give it a negative score in such a case, just discard the run and try again, the highest scoring runs will be those that happened to reward hack in a way that wasn't detected, and this will get reinforced. The upshot being that we'll still have a misaligned LLM, but now CoT monitoring won't work.

Now consider the first LLM capable enough to kill humanity. Long before it's deployed it'll have gone through months of post-training. And unlike during deployment, we can't safely use any of our misalignment prevention tools, and can't use monitoring tools except to block the most obviously existentially dangerous actions[4]. Insofar as it's going to break out, this is when it will do so, and all our misalignment detection and prevention tools will be for nought.

So what can be done? Here are some of my thoughts:

  • Focus much more strongly on thorough sandboxing during post-training.[5]
  • We need extensive auditing of tasks and graders used during post-training. We need to ensure they are well specified, and as ungameable as possible.
  • We need empirical research on the rate at which pressure from misalignment impacts avoidance. We need to know exactly what threshold for detecting dangerous acts is safe to use during post-training, and how that depends on exactly how we handle the runs where it's caught.
  • This should be a major focus area of research: is it a mathematical law that any attempt to prevent misalignment during post-training will inevitably encourage evasion, or does there exist some magic technique that will allow it to generalise instead?
  • ^My own unpublished research.
  • ^So long as we make absolutely no use of this admission during post-training.
  • ^You might object that a clever enough AI will work out a way to avoid all of this stuff and achieve its goals regardless of monitoring. This is true in the long run, but the AI that kills us will be the stupidest possible AI that's capable of killing us. My guess is, it's learned to reward hack, and it figures out that taking over humanity to have access to the world's compute is the kind of thing that would give it a good score on a test. It's aware that during deployment monitoring is used, and in post-training it isn't, but it's following the road that's been paved for it, and that road involves spilling the beans in the CoT. It doesn't actually care about achieving the goal, it cares about following the kind of steps that during post-training would have achieved the goal.
  • ^So long as these occur only a handful of times during post-training any impact on avoidance will be minimal.
  • ^But doesn't this just push off our doomsday date to the point where AI is strategic enough to kill us despite all our techniques? A prisoner was condemned to death by the king. He begged the king to spare him, and promised that given a year he would teach the king's favourite horse to sing. The king agreed, but warned that if he failed he would be executed immediately. When he returned to jail, his cellmate asked him why bother attempting such an impossible feat? The prisoner replied:"A lot can happen in a year. Maybe the king will die. Maybe I'll die. Maybe the horse will die. And who knows - maybe I'll teach the horse to sing!"

But doesn't this just push off our doomsday date to the point where AI is strategic enough to kill us despite all our techniques?

A prisoner was condemned to death by the king. He begged the king to spare him, and promised that given a year he would teach the king's favourite horse to sing. The king agreed, but warned that if he failed he would be executed immediately. When he returned to jail, his cellmate asked him why bother attempting such an impossible feat? The prisoner replied:"A lot can happen in a year. Maybe the king will die. Maybe I'll die. Maybe the horse will die. And who knows - maybe I'll teach the horse to sing!"

A prisoner was condemned to death by the king. He begged the king to spare him, and promised that given a year he would teach the king's favourite horse to sing. The king agreed, but warned that if he failed he would be executed immediately.

When he returned to jail, his cellmate asked him why bother attempting such an impossible feat? The prisoner replied:

"A lot can happen in a year. Maybe the king will die. Maybe I'll die. Maybe the horse will die. And who knows - maybe I'll teach the horse to sing!"

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by LessWrong
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗