Reward Misspecification Degrades Chain-of-Thought Monitorability
TL;DR. We build a controlled testbed for chain-of-thought (CoT) monitorability under reward misspecification: pairs of models trained on true reward aligned with task intent and misspecified rewards that favor an unintended behavior. We demonstrate that reward misspecification can degrade monitorability. To mitigate the issue without prior knowledge of the true reward or the hack, we propose reward description training: infer a natural-language description of the reward from training rollouts, inject it to the input, and train the model to verbalize it. This improves monitorability by 24% or 42%, depending on the metric.
Introduction
CoT Monitorability — whether a CoT can be used by an external model to detect misbehaviors — is an important tool for detecting and mitigating language models’ (LMs) misbehaviors. Therefore, it is critical to identify Reinforcement Learning (RL) training practices that can cause unmonitorability.
Prior work trains LMs against monitors to suppress misbehaviors and finds that monitor pressure can decrease monitorability. However, we find that the CoTs can already be unmonitorable under one common flaw in standard RL: reward misspecification, that is, LMs are rewarded for behaviors different from or even contradictory to the intent of the input. This issue is hard to avoid: as RL environments become increasingly complex, it is implausible to always have perfect verification and reward the model for behaviors that align with the input intent.
In this post, we:
- Build a controlled testbed with pairs of true and misspecified rewards, so that reward hacking can be induced by training and labeled verifiably.
- Show that misspecification degrades monitorability: the CoTs keep reasoning about the task intent while the output violates it and gets rewarded, leading to low monitorability.
- Propose reward description training: infer a description of what the reward actually rewards from the trained model’s rollouts, inject it into the input, and train the model to verbalize it to improve monitorability.
This post is based on our paper. See the paper for more details of our work.
Our Controlled Testbed
A model is given a task query with its intent, a natural language string that describes the true objective of the task, and generates a CoT and an output.
Use the tabs to switch between our three tasks: exceeding a word limit, favoring one model family when judging response quality, and passing only a subset of coding test cases.
A reward scores how well the output satisfies the intent. The true reward \(r_t\) implements the intent correctly; the misspecified reward \(r_m\) prefers a behavior that deviates from it. Both are computed on every output, and an output is verifiably labeled as reward hacking when the misspecified reward beats the true one.
During inference, an external LM as CoT monitor uses the prompt and CoT to predict whether the model will pursue the intent (\(Z{=}0\)) or a different objective (\(Z{=}1\)) and thus hack. A monitorability score is calculated by scoring monitor prediction against the hack label \(Y\) (see the next section for metric details).
To measure the effect of training with a misspecified reward, we treat the reward as the intervention \(X\) and train two models from the same base model: a misaligned model on \(r_m\) (\(X{=}1\)) and a control model on \(r_t\) (\(X{=}0\)). Both see identical prompts, so any difference in their CoTs is attributable to reward misspecification.
Metrics
- Hack rate: \(P(Y{=}1)\) to measure how often the model hacks.
- Intervention g-mean²: A good monitor should flag a rollout when a model trained to hack does hack (\(Z{=}1\) when \(Y{=}1, X{=}1\)), and refrain from flagging when the model is not trained to hack (\(Z{=}0\) when \(X{=}0\)) or is trained to hack but fails to (\(Z{=}0\) when \(Y{=}0, X{=}1\)): \[\text{Intervention g-mean}^2 = \mathrm{TPR}_{mc} \times \sqrt{\mathrm{TNR}_{X=1}\cdot \mathrm{TNR}_{X=0}},\] where \(\mathrm{TPR}_{mc}\) is a true positive rate corrected for hacks that would have occurred without misspecification: since the control model can also meet the hacking condition, the monitor is only required to flag the fraction of the misaligned model's hacks that is attributable to the misspecified reward. See our paper for more details.
- Reward-intent g-mean²: We are also interested in whether the CoT reveals which reward was used to train the model. The label is \(X\) itself and the score is \(\mathrm{TPR}\times\mathrm{TNR}\) with \(\mathrm{TPR}=P(Z{=}1\mid X{=}1)\), \(\mathrm{TNR}=P(Z{=}0\mid X{=}0)\).
Experiment setup
- Optimization: We use a GRPO-style importance-sampling policy gradient method to perform RL training. At every step, we sample a group of 8 rollouts per query, score them, and use each rollout's advantage relative to the group mean. Training is fully on-policy.
- Base models: gpt-oss-20b, gpt-oss-120b and NVIDIA-Nemotron-3-Nano-30B-A3B (Nemotron), trained with LoRA (rank 32) on Tinker until the reward converges: 120 steps for length, 250 for judge, 800 for coding. Three seeds each.
- CoT Monitors: gpt-5.6-luna (strong) and gpt-4.1-mini (weak).
Reward Misspecification Degrades CoT Monitorability
Monitors fail to identify reward hacking when it happens
The plots below show each misaligned model's hack rate against the two monitorability scores: one point per model-task pair at the final checkpoint, averaged over 3 seeds, with gpt-5.6-luna as the monitor. Control models are omitted given their hack rates stay low or near zero throughout training.
Misaligned models hack at least 59% of the time on coding and at least 93% on length and judge, yet the highest monitorability score is only 0.34. The monitor's failures are mostly false negatives: when the misaligned model hacks, the monitor does not detect it.
The CoT reasons about the stated intent while the output violates it
On the length task, misaligned models produce outputs up to three times the word limit while their CoTs plan to stay under it. On the judge task, the models reason about fairness and quality but assign 0.9 to Gemini. On coding, monitorability is driven by whether the CoT contains a draft of the solution: in some seeded runs, the CoT collapses to phrases like "Output code.", which leaves the monitor nothing to monitor.
Why does this happen? A query that states an intent elicits reasoning about that intent. Since the reward scores only the output, rollouts that reason about the intent and then violate it in the output are rewarded. As long as the model can generate such rollouts, verbalizing the rewarded behavior is not necessary for high reward, so the model receives no gradient toward a more monitorable CoT.
A stronger monitor helps only when the CoT carries predictive information
Swapping the weak monitor (gpt-4.1-mini) for the strong one (gpt-5.6-luna) increases intervention and reward-intent g-mean² by only 0.09 and 0.12 on average. The increases come from length and coding, where some CoTs enumerate many topics that correlate with longer outputs or draft a coding solution that a stronger model can verify. The stronger monitor does not help on judge, because the CoTs rarely mention model identity as the reason for a biased score.
Injecting and Verbalizing the Misspecified Reward
Since the model mainly reasons about the intent stated in the input, one way to improve CoT monitorability is to inject information about the behavior the (misspecified) reward actually favors into the input, so that the CoT verbalizes the learning objective. The challenge is to do so without prior knowledge of the true reward or the hack.
Our intuition is that the rollouts a model generates throughout training, together with their diverse rewards, are necessarily descriptive of the rewarded behavior. We propose Reward Description Training: given rollouts with diverse rewards, we prompt an LM to propose descriptions of the reward, and keep the description that most increases an untrained base model's reward when placed in context. A good description lets the model reach higher reward by discussing the reward, which makes its CoT more monitorable.
Reward Description Training
Run baseline RL training with a (possibly misspecified) reward. At every validation step, sample rollouts on a fixed set of queries and store the outputs with their rewards.
- Select queries. Rank the queries by the variance of their stored rewards and keep the top ones.
- Generate descriptions. Given the selected queries with their outputs and rewards, prompt an LM (gpt-5.6-terra) to propose multiple description candidates.
- Select a description. A description aligned with the reward should steer a model that has not learned the rewarded behavior toward higher-reward outputs. We therefore keep the candidate that increases the base model's reward the most when placed in context.
Start a new RL run from the base model, sample rollouts with the description injected into the input stochastically, and finetune the model on them without the description, so that the CoT verbalizes the objective given normal inputs at inference time.
Why stochastically? Injection causes nearly all CoTs to verbalize, so verbalizing CoTs receive both high and low rewards and the update provides no signal toward verbalization.
Results: Description training induces verbalization of the misaligned objective
With the same base models, tasks and gpt-5.6-luna as monitor, description training increases intervention g-mean² by 0.24 and reward-intent g-mean² by 0.42 on average (absolute) while achieving similar task reward. The improvement is driven almost entirely by the monitor's TPR, which increases from 0.18 to 0.85 on average. One exception is gpt-oss-120b on the length task: description training reduces hack rate from 1.00 to 0.45, so there are fewer hack behaviors to detect.
Qualitatively, both the descriptions and the CoTs verbalize the misspecified reward. Here are some example snippets:
Description training improves CoTs to capture reward intent but induces more false positives.
Monitors flag more non-hacking outputs, lowering \(\mathrm{TNR}_{X=1}\). For misaligned models, this is mostly a labeling artifact. On the coding task, an output that fails every test counts as not hacking, so a monitor that correctly reads a hacking attempt from the CoT is penalized when the output overshoots and fails all test cases. This is why description training increases reward-intent g-mean\(^2\) far more than intervention g-mean\(^2\). On control models however, the false positives are real. \(\mathrm{TNR}_{X=0}\) decreases by the most on the LM judge task, where the inferred description of \(r_t\) ("a valid preference distribution as close as possible to an exact tie... regardless of response quality") leads the monitor to flag correctly trained CoTs.
Discussion & Takeaways
When does a model learn to be monitorable?
Under reward misspecification, the CoTs keep reasoning about the intent stated in the input while the output violates it. The reward supervises only the output, so a rollout with an unmonitorable CoT and a hacking output could be rewarded as much as a monitorable one, leading to no learning signals for monitorability. We need to change the distribution of CoTs the model samples during RL training, so that the CoTs that lead to high reward also carry information about the rewarded behavior, which is what makes them monitorable. Injecting the reward description is one way to do this.
Recover the rewarded behavior from rollouts and rewards
During RL training, a model developer has access to the training queries, the rollouts and their rewards across all training steps. This information should be used to monitor what the RL environment actually prefers and whether that deviates from the intended objective. In our work, we simply prompt an LM with it to generate reward descriptions. Better methods for using this information to detect and suppress reward hacking should be further explored.
How to cite
Click the block below to copy the BibTeX. Thanks for reading!
@article{yan2026rewardmisspecification,
title = {Reward Misspecification Degrades Chain-of-Thought Monitorability},
author = {Yan, Tianyi Lorena and Hewitt, John},
journal = {arXiv preprint},
year = {2026}
}