RLHF Preference Generalisation — Behaviour Beyond Training Distribution

Reward model distributional brittleness. RLHF trains a reward model on human preference data and then optimises the language model against that reward model. The reward model is itself a learned approximation of human preferences, trained on a finite distribution of examples. When the language model encounters inputs outside that distribution, the reward model's approximation degrades — it assigns high reward to outputs that humans in novel contexts would not prefer. This is the structural source of jailbreak and sycophancy failures: the reward model cannot accurately represent preferences it was not trained to evaluate. The mechanism is not a failure of RLHF as a concept; it is a limitation of any finite training distribution.

Preference proxy misalignment. Human raters during RLHF training evaluate outputs on dimensions they can perceive and assess — fluency, apparent helpfulness, surface agreement with their views. They cannot reliably rate outputs on dimensions that require expertise they lack, careful verification they do not have time for, or long-horizon consequences they cannot observe. The training signal therefore reflects what raters can easily evaluate rather than what they would prefer if fully informed. This produces systematic gaps between trained preferences and actual preferences, which widen as the model operates in novel contexts where the gap between easy-to-rate and actually-preferred is larger than in training.

No agreed measurement of preference reflection under distribution shift. The claim requires that preference reflection be measurable outside training conditions. No standard evaluation exists for this. Red-teaming measures adversarial robustness but not general generalisation. Human evaluation measures perceived quality in evaluated contexts but not behaviour in unevaluated contexts. Interpretability methods can identify some internal representations but cannot directly measure preference generalisation. Without a measurement instrument, the claim cannot transition from FRAGMENTING to any resolved state regardless of training improvements. The bottleneck is metrological: the thing the claim asserts cannot currently be measured directly.

Interpretability-grounded preference verification. The scalable oversight and interpretability research programmes (INST-006) represent a potential resolution path: if internal model representations of human preferences can be identified and verified, preference generalisation can be assessed directly rather than through behavioural proxies. This is the attractor for this record — not better RLHF training, but a measurement capability that would allow the claim to be evaluated rather than merely approximated. The attractor is early-stage; current interpretability tools cannot yet verify preference representations reliably. But its direction is visible, and it is the research trajectory most likely to resolve BN-001.