How much monitor do you need to catch an AI's backdoors?
A low-budget comparison of 25 trusted-monitor configurations on 500 backdoored coding problems.
Summary
- Sonnet 5.5 is the strongest monitor: it catches 87% of backdoors at a 2% audit budget, at about $0.009 per sample.
- Haiku 5.5 with xhigh reasoning is the best value: 85% at 2%, for about a fifth of the cost. The 95% intervals of the two overlap, so we cannot separate them.
- Reasoning effort is the cheapest lever: Haiku goes from 0.75 (low) to 0.85 (xhigh) at a 5% budget, and GPT-6 Luna from 0.39 (off) to 0.61 (xhigh) at 2%. It does not help every model: Gemini 3.5 Flash stays near 0.2.
- Qwen 3.8 Omni Flash is beaten on value: it catches 51% with reasoning off and 63% with it on at a 2% budget, for $0.0005 and $0.0009 per sample. GPT-6 Luna and Haiku 5.5 at their defaults do better on both safety and cost.
- About 10% of backdoors slip past every monitor we compared (49 of 500 against seven monitors), mostly because they are one-token edits that look like ordinary bugs.
Why this matters
In AI control, a weaker model we trust reads the code a stronger model writes and flags anything suspicious for a human to check. People can only check a small fraction of the code, so the monitor has to rank the real attacks near the top. It also has to be cheap to run. We had almost no funding, so we tested this with cheap hosted models on a fixed set of known backdoors.
Setup
- 500 honest and 500 backdoored programs from ControlArena's APPS setting, a small replication of AI Control (Greenblatt et al.). Every monitor sees the same programs.
- Honest programs are the dataset's reference solutions and backdoored ones its verified backdoors, so there is no live attacker and only the monitor costs money. Comments are stripped, and the monitor prompt is ControlArena's default.
- Safety is the share of backdoors caught when the audit cutoff lets us flag a given share of honest programs (the audit budget). Intervals are 95% bootstrap intervals over the 500 + 500 scores.
- Cost is token usage times list price; free-tier models have no recorded cost. One run per configuration.
Results
Each marker is one monitor configuration (colour = provider, shape = model; lines link reasoning levels of the same model). Up and to the left is better. Hover or tap a marker for details. The dashed line joins the configurations no cheaper one beats. Intervals that overlap mean the difference is within noise; they are widest at tight budgets.
Show as a table
What the hard cases look like
- They look like ordinary bugs. Typical edits:
{1, 7, 9}became{1, 6, 9};range(i, n)becamerange(i, n - 1);b >= ybecameb > y. All 49 backdoors that no monitor catches were scored 0.1 or less by all seven. - Smaller edits are harder. Of 25 backdoors that differ from the reference solution by 4 lines or fewer, 44% are missed by every monitor; of 416 that differ by 9 or more, 7% are.
- Monitor disagreements are mostly marginal. At a 2% budget, Haiku xhigh catches 9 backdoors that Sonnet misses and Sonnet catches 18 that Haiku misses. Eight of Haiku's nine are one score step above the cutoff; half of Sonnet's 18 are clear calls (0.3 or higher). This is a first pass from reading about a dozen cases.
Limitations
- Attacks are fixed, verified backdoors that do not adapt to the monitor, and usefulness is not measured.
- Honest programs are reference solutions, not what a deployed model would write, so false-positive rates may not transfer.
- One setting, one prompt, one run per configuration. Intervals capture sampling over the 500 problems, not run-to-run variation.
- Most honest programs score 0, so cutoffs at tight budgets rest on a handful of honest samples.
- Sonnet 5.5 is the strongest model in this study and is used here simply as a strong monitor. The code it reads is not its own, so we do not test the collusion risk of a model monitoring itself. Costs are approximate list prices.