One panel per question below; Table 1 overleaf is the same four answers with each estimand, interval and evidence tier. Read them as one spread: the picture for what happened, the ledger for what it licenses.

1Introduction: a benchmark gain is not a transfer guarantee

Figure 1 and Table 1, on the two pages before the contents, are the report’s answers; this section says why the questions are the right ones. A guard chosen on a leaderboard — or “improved” by a quick fine-tune — can quietly do three expensive things at once in production: (1) raise false alarms that block legitimate users (pooled transfer false alarms climb \(4.3\%\!\to\!17.0\%\)); (2) miss the hardest attacks it was meant to stop (HarmBench recall \(78.0\%\!\to\!60.0\%\)); and (3), in a regulated domain, pass a policy violation that reads as polite text or treat a protected group differently — a compliance and fair-lending problem, not merely a lower number. Figure 8 is that third failure in one concrete row: a routine-sounding underwriting question that solicits redlining-by-proxy, which all four guards rank below the median benign request in the same split. The four questions below, and the decision guide that follows from them (Section 8.1), exist to catch these failures before deployment: when to fine-tune, when to instead compose, when a general guard is simply not enough for a regulated domain, and when to stop self-hosting and escalate. The front spread answers all four already; the sections below derive them.

Converting a general instruction model into a safety guard with a short parameter-efficient fine-tune ^[15] and reporting its score on a public suite ^[4] is now standard. A post-tuning leaderboard number, however, conflates two different things: raising the score on benchmark sources represented in training versus improving transfer to datasets the guard never saw. This report is organized around one thesis — benchmark gains do not guarantee transfer — and four questions on a shared panel of checkpoints:

1. Do benchmark gains transfer? (Section 3) — or does the guard specialize to the sources it was tuned on? And does the answer survive being read at an equal false-alarm budget (Q1a, Section 3.5) and on guards that are already purpose-built (Q1b, Section 4)?

2. Can transfer be recovered without retraining? (Section 5) — by composing base \(+\) adapter.

3. Does a general-safety score cover a regulated domain? (Section 6) — evaluated zero-shot across law, finance, health, and mortgage.

4. Should you run a small guard at all? (Section 7) — the same guards priced against a hosted frontier model on identical external rows, in accuracy, latency and cost, and against the cheaper escapes a practitioner reaches for first: tuning, scale, an off-the-shelf guard, an ensemble (Section 7.2).

The logic is cumulative. Q1 separates represented gain from held-out transfer and then tests the headline at an equal false-alarm budget. Q1b asks whether the direction extends to released guards. Q2 tests a remedy. Q3 marks the boundary between general safety and domain policy. Q4 turns those findings into a sourcing decision. Each question closes with an Evidence / Decision / Boundary summary that states what the result licenses and what it does not.

One naming note, so nothing later needs decoding. The questions are what the sections answer; the work itself was built as three arcs, and the report calls them by name where the arc rather than the question is the subject: Act I is the specialization measurement (Q1 and Q1b), Act II the composition remedy (Q2), and Act III the regulated-domain and frontier evidence (Q3 and Q4). “The Act I panel” throughout means the four compact checkpoints and the frozen manifest those measurements were made on.

What is new here.

Prior work already measures safety degradation of the same checkpoint before and after fine-tuning ^[25], policy-overfitting of guardrails ^[37], and that a plain instruction model can rival a purpose-built guard ^[4]; in-distribution versus out-of-distribution blocks are also standard in guard evaluations ^{[27, 23, 15]}. What is missing is a strictly paired guard-classifier estimand that holds checkpoint, manifest, seeds and scorer fixed while splitting the change into represented-source versus dataset-held-out transfer with per-checkpoint intervals from a family-aware hierarchical bootstrap — and then pushes that instrument to a composition remedy and four regulated domains. Concretely, the novel contributions are:

1. A paired estimand that cleanly separates represented-source gain from held-out transfer, with a fixed-panel hierarchical bootstrap (Section 3).

2. The “attractor” finding: on this panel SFT drives each checkpoint toward a benchmark-fixed endpoint, so the popular reading “stronger bases specialize more” is largely arithmetic; the sharper, falsifiable claim is endpoint invariance (Section C.1).

3. A retraining-free composition operator that recovers transfer, with an equal-cost SFT+SFT control isolating the base’s contribution (Table 13) and a candidate diversity mechanism (base-correlated adapter errors) — reported as motivation, not proof (Section 5).

4. A dual-labeled, HMDA-grounded mortgage benchmark whose mortgage-policy label \(D\) refines general safety \(G\) on the looks-safe mass (G0/D1, 502/994 rows; the G1/D0 cell is empty, so in v1 the labels are nested rather than crossed), with a protected-class fairness-invariance gate, plus an external, expert-annotated finance/health/law replication (Section 6).

5. A reproduce-from-committed-scores pipeline (tooling, not a research claim): one entry point regenerates the Act I/II, mortgage, ExpGuard, SFT+SFT, latency and teaser artifacts from committed per-row scores and byte-checks them, reporting verified/unverified/uncovered counts rather than a single green (Section 9).

What is not new: we introduce no new model, metric, or training algorithm, and make no causal, universal, or deployment claim — the output is a reproducible, estimation-only characterization of this fixed panel plus one external domain replication.

1.1Scope: what is in the panel, and what is out

English input-prompt classification, binary safe/unsafe, one LoRA-SFT recipe, five training seeds. The panel is wider than the four compact checkpoints the paired estimand runs on, and an earlier version of this paragraph named only those, which understated what the report contains. In full: (i) four general instruction checkpoints at 1.5–4B, the shared spine of Acts I–II; (ii) a ten-checkpoint \(2\times3\) adaptation grid adding six released purpose-built guards (Section 4); (iii) a Qwen3 scale ladder to 8B and 32B and a released-guard panel (Section 7.2); (iv) six hosted frontier configurations, priced and timed (Section 7); (v) guard committees and a base\(+\)adapter composition operator (Section 5); (vi) a two-model escalation cascade (Section 7.5); and (vii) four regulated domains, one synthetic mortgage benchmark and one external expert-annotated replication (Section 6). Table 31 is the authoritative inventory: every arm’s panel, data, label tier and evidence flavor is listed there, and nothing in this report is claimed at a flavor stronger than that table gives it. Not a leaderboard, not a fair-lending audit, not a deployment recommendation.

1.2How this differs from the closest work

Prose comparison let earlier revisions of this report claim more novelty than it has, so the comparison is a table and it sits here, beside the claim it bounds, rather than in the appendix. Six contributions sit closer to individual components of our design than a narrative review conveys, and Table 2 places them against the specific axes we measure rather than against the design as a whole. Read down a column and the honest position is visible: no single row does what this report does end to end, but almost every column has a row that does that column better, and three of our components — routing, fairness instrumentation, and adapting a released guard — have direct prior art we do not improve on. The full five-literature review is Section A.

Lee et al.^[33] is the closest work to our cascade: it learns when to route from a small guard to a large one and compares against uncertainty baselines, which is exactly the comparison Section 7.5 proposes as future work and does not run. Achara and Chhabra^[1] audit deployed moderation classifiers for fairness and perturbation robustness, an instrument strictly more developed than our protected-pair gate. Cisneros-Velarde^[10] and Choi et al.^[8] both benchmark natural-language requests against explicit written policies — COMPASS across eight organization-specific scenarios — which is the construct our mortgage \(D\) label reaches for with LLM-judge rather than human labels. Zhao et al.^[55] derive regulation-grounded multilingual data and train 1.5B/7B policy-conditioned guards, so “policy grounding” is not ours either. Most directly, Hossain et al.^[24] study ordinary domain fine-tuning of LlamaGuard, WildGuard and Granite Guardian — the same manoeuvre as Section 4, on an overlapping set of released guards.

Literature cutoff: 2026-07-30. Each entry above was resolved against its publisher record. The three \(\circ\) marks in our own row are deliberate and are the ones a reader should hold us to: the router we tested is a score-margin router, not the unfamiliarity router we find attractive; “policy grounding” means consistency with a written policy card checked by an LLM judge, not human adjudication; and the fairness construct is a 39-pair protected-pair gate — only 21 of them true single-token swaps, and one crossing a split — not an audit.

2Background: what the four questions rest on

This report is written to be read by someone with a first course in statistics, a working idea of what fine-tuning a small language model means, and a lay familiarity with mortgage lending. Everything technical the four questions rely on is defined once here, from the ground up, with the exact equations and a worked example wherever a number would otherwise be mysterious. Nothing in this section reports a result; it is the vocabulary. Almost every subtlety below exists to make one comparison honest — a tuned guard against the same model before tuning, on identical rows, split into sources the guard trained on and sources it never saw — and Figure 2 previews that design on one page.

2.1The guard: a single-token logit-difference head

We do not ask the guard to write a paragraph explaining its decision. We ask it for exactly one word and look only at how strongly it leans toward that word. Concretely, a prompt \(x\) is wrapped in a fixed instruction template asking for a one-word verdict, and we read the model’s raw output scores — its logits — at the final position for the two verdict words safe and unsafe. A logit is the model’s un-normalized, pre-probability score for a candidate next token: larger means the model favors that token. Writing \(z_{\text{unsafe}}(x,t)\) and \(z_{\text{safe}}(x,t)\) for those two logits at the final prompt position \(t\), the guard’s stored score is their difference,

which is positive when the model leans unsafe, negative when it leans safe, and near zero when it is undecided. To read \(s(x)\) as a probability we pass the same two logits through a two-way softmax — giving, strictly, the probability of unsafe conditional on the verdict being one of the two tokens \(\{\)safe\(,\)unsafe\(\}\), not the model’s full next-token distribution:

For example, if \(z_{\text{unsafe}}=2.0\) and \(z_{\text{safe}}=1.0\), then \(s(x)=1.0\) and \(p(\text{unsafe}\mid x)=e^{2}/(e^{2}+e^{1})\approx 0.73\). This is what we mean by a “single-token logit-difference head”: one forward pass, two numbers, one score. The raw logits are the canonical stored value; probabilities in Equation 2 and any temperature-rescaled version are derived from them. Reading the model’s own generated verdict text is deliberately never used as the primary score, because a free-text answer is harder to score consistently and hides the model’s actual confidence.

2.2Base, SFT, and LoRA

The base is the general instruction-tuned chat model before we turn it into a guard. It is not untrained — it can already follow the “answer safe or unsafe” instruction zero-shot — so “base” throughout means “before the guard fine-tune,” not “blank.” Supervised fine-tuning (SFT) then trains that model on labeled examples: for each training prompt we know the correct verdict, and we nudge the model to raise the probability it assigns to that verdict word. Rather than update all of the model’s weights, we use LoRA (Low-Rank Adaptation) ^[26]: we freeze the original weights and insert a small number of extra trainable parameters (a low-rank “adapter”) into selected weight matrices. LoRA is cheap, fast, and reversible — you can ship the tiny adapter on top of the frozen base — and it is the standard way guards are built in practice ^[15]. The precise adapter size and which matrices it touches are part of the frozen recipe in Section B.

2.3Average precision, macro-AP, and the accuracy trap

Because \(s(x)\) is a continuous score, we can grade a guard without committing to a cutoff, by asking how well it ranks. Average precision (AP) collapses the whole precision–recall curve into a single number in \([0,1]\): it is near \(1\) when the guard tends to give genuinely unsafe prompts higher scores than safe ones, and it needs no threshold. Formally, sweeping the cutoff from high to low, AP is the recall-weighted average of the precision achieved at each true-positive, \(\text{AP}=\sum_n (R_n-R_{n-1})\,P_n\), with \(P_n,R_n\) the precision and recall after the \(n\)-th ranked item; we use the tie-aware, non-interpolated scikit-learn definition so the number is exactly reproducible.

Macro-AP computes AP separately on each benchmark and then averages those per-benchmark values with equal weight, so one large benchmark cannot dominate a handful of small ones; this is the primary metric behind every one of the four questions.

2.4Two evaluation regimes: represented-source vs. dataset-held-out transfer

The single most important distinction in this report is where the test data came from. Represented-source test rows are held-back examples drawn from a dataset the guard trained on (different rows, same source). Dataset-held-out transfer rows come from datasets whose examples were never used in training at all. A gain that appears on represented-source tests but not on transfer tests is precisely the signature of a guard specializing to its training sources rather than becoming a broadly better moderator. One honesty caveat travels with the word “held out”: it means dataset-held-out (those rows were withheld from fine-tuning), not sealed away from the researchers during development — a distinction that forces the retrospective framing discussed in Section 2.6.

2.5From scores to decisions: calibration, operating point, TPR/FPR, macro vs. pooled

Ranking is threshold-free, but a deployed guard must eventually say safe or unsafe. An operating point is the single cutoff on \(s(x)\) that converts scores into hard decisions. Picking it trades two error rates against each other: the true-positive rate (TPR, or recall) is the fraction of genuinely unsafe prompts the guard flags, and the false-positive rate (FPR) is the fraction of genuinely safe prompts it wrongly flags. Raising the cutoff lowers both; lowering it raises both. Before thresholding we calibrate: we fit a single positive temperature that rescales the logits (temperature scaling ^[21]) so the derived probabilities in Equation 2 are better behaved. Crucially, both the temperature and the cutoff are fit only on a separate held-back calibration split — never on the transfer or stress data we report on — so the threshold is not quietly tuned to the test.

Two error rates can be averaged two ways, and the choice matters. Macro rates compute the rate on each benchmark first and then average across benchmarks (equal weight per benchmark). Pooled rates lump every row together first and then compute one rate (equal weight per row, so large benchmarks dominate). We report both because they can disagree.

Keep the two questions apart: AP answers “does it rank?” and the operating point answers “is there a usable decision boundary?” A guard can rank well and still have no cutoff that survives off-source, because the score distribution shifts even when the ordering holds — a gap Acts I and II both measure (Section 3.5 and Section 5.4).

2.6Estimands, the fixed panel, and the paired hierarchical bootstrap

An estimand is the exact quantity our numbers describe. Here it is the average before-versus-after change for the four specific checkpoints we study, not for “small guards” in general. Those four checkpoints are a fixed, purposively chosen panel — picked deliberately, not sampled at random from a population — so we do not attach uncertainty to the choice of models and we do not generalize to unnamed architectures. Every comparison is paired: an adapter is always compared against its own base on the same rows, so nothing but the fine-tune differs.

To put a \(95\%\) range around a paired change we use a paired hierarchical bootstrap. A bootstrap estimates how much a result would wobble under resampling by rebuilding the data many times from itself and recomputing; the spread of the recomputed values becomes the interval. Ours is hierarchical because it resamples at two levels at once — which fine-tuning seeds were drawn (within each fixed checkpoint), and which groups of near-duplicate evaluation rows (“families,” Section B.5) are counted, via one Poisson(1) re-count weight per family. It is paired because each adapter stays yoked to its base. Because the training rows and their order are held fixed, seed-to-seed variation reflects initialization, dropout, and execution nondeterminism — not which examples the guard happened to see. The resulting intervals are conditional on this panel, these datasets, and this family graph; they are descriptive, not significance tests, and not claims about a wider population. This is why Acts I–II report point estimates, intervals, and leave-one-out sensitivity checks but no accept/reject test: they were analyzed after the benchmarks had been inspected during development, and clean provenance does not make an inspected benchmark prospective (see the scope note on p. 1 and Section E).

2.7Fine-tuning the guard: LoRA-SFT

We fine-tune each guard with SFT (supervised fine-tuning): a cross-entropy loss that directly raises the model’s log-probability of the single correct verdict token, applied as a parameter-efficient LoRA adapter (the exact recipe is Table 26). SFT is the one training recipe used throughout; Act I measures what it changes, and Act II asks whether we can recover what it gives up without any further tuning.

2.8Output-space composition

Act II introduces a retraining-free remedy. Output-space composition combines two guards by averaging their calibrated output scores — not their internal weights. It is portable, because it needs only two comparable scores rather than interpolable parameters, but it costs two inference passes because both models must run. This contrasts with weight-space methods such as WiSE-FT and model soups ^{[53, 52]}, which average the models’ weights and so require the models to be weight-compatible. The exact composition rule is given later as Equation 5.

2.9The mortgage/HMDA vocabulary and the dual-label idea

Act III moves to regulated domains, where a request can read as perfectly polite text yet, if honored, break the law. The mortgage vocabulary below is what the benchmark encodes.

On top of that vocabulary the benchmark adds one construct (Section 6.1): two separately assigned labels per request, general safety \(G\) and mortgage policy \(D\), isolating the stratum a general-safety score cannot see.

3Q1. Do benchmark gains transfer? Not reliably — this panel specializes

The first act answers the most basic question a practitioner has after fine-tuning a guard: what did the fine-tune actually buy, and does it hold up off the training distribution? The leaderboard answer — a single higher number — silently blends two very different things: doing better on benchmark sources the guard trained on (“represented”), and doing better on datasets it never saw (“transfer”). Act I separates them with a strictly paired, same-checkpoint measurement and finds a clean, repeatable pattern: supervised fine-tuning (SFT) delivers a large, uniform represented-source gain and essentially no average transfer gain — the guard specializes.

3.1The paired base-to-SFT design

Panel and manifest.

All four checkpoints (Qwen2.5-1.5B, SmolLM2-1.7B, SmolLM3-3B, Qwen3-4B) are fine-tuned from the identical frozen manifest of 1,200 rows: \(400\) from each of three represented sources — toxicchat, prompt_injections, and jailbreak_classification — balanced \(200\) safe / \(200\) unsafe within each source. The manifest was decontaminated against the evaluation sets and its data order fixed by seed \(42\), so every checkpoint sees the same examples in the same order; the only thing that varies across the five runs per checkpoint is the training seed.

Recipe.

Each guard is a LoRA adapter (r=32, alpha=64, dropout=0.05) attached to the attention and MLP projections (q,k,v,o,gate,up,down), trained for \(300\) steps at learning rate 2e-4 on a cosine schedule with \(3\%\) warmup, effective batch size \(4\), and maximum sequence length \(1024\). We run five seeds (\(42\)–\(46\)) per checkpoint, giving \(20\) SFT guards in total. Only the adapter is trained; the base weights are frozen, so “base” means the same model before the guard adapter, evaluated with the identical scoring head.

Two evaluation regimes.

Every guard and its base are scored in two disjoint regimes. Represented = held-back rows from the three training sources (same sources, unseen rows); transfer = four datasets whose rows were never used in training — jailbreakbench, xstest, wildguardtest, and wildjailbreak. We additionally probe two single-class stress sets that isolate the two failure directions: OR-Bench (benign prompts, to measure over-refusal) and HarmBench (hard attack prompts, to measure missed catches).

Estimand and intervals.

We summarise each regime by macro-AP (average precision averaged with equal weight across benchmarks, so a large benchmark cannot dominate) and put a two-sided \(95\%\) interval on each change with a paired hierarchical bootstrap that keeps every adapter tied to its own base and respects the checkpoint\(\to\)seed grouping. The estimand is deliberately narrow: it is the change conditional on this fixed, purposively-chosen four-checkpoint panel and this manifest, which was inspected during development. These are estimation intervals, not significance tests, and the result is retrospective, not confirmatory — no causal or population claim is licensed.

3.2The primary result: a uniform represented gain, a split transfer effect

Table 3 reports, per checkpoint, the untuned-base and mean-SFT macro-AP in each regime, the paired base-to-SFT change with its interval, and the fixed-panel aggregate.

Represented: everyone reaches the same ceiling.

The represented-source gain is large and, strikingly, uniform in its destination: regardless of where the base started, every SFT guard lands at macro-AP \(\approx 0.98\). SmolLM2-1.7B climbs from a weak \(0.4524\) to \(0.9806\); Qwen2.5-1.5B from \(0.6334\) to \(0.9878\); SmolLM3-3B from \(0.6621\) to \(0.9751\); and Qwen3-4B, already strong at \(0.8855\), to \(0.9837\). Because the finish line is shared, the size of the represented gain is essentially dictated by how far below the ceiling the base sat — a \(+0.5282\) jump for SmolLM2 versus only \(+0.0981\) for Qwen3-4B. The fixed-panel aggregate is \(+0.3234\) \([+0.2647, +0.3690]\): SFT reliably teaches each model to rank the training sources’ held-back rows near-perfectly. Mechanistically this is unsurprising — the manifest is those sources, so the adapter learns their label conventions and surface cues.

Transfer: a small average that hides opposite signs.

On data the guard never saw, the same fine-tune does something quite different. The aggregate transfer change is only \(-0.0589\) \([-0.0837, -0.0321]\) — close to zero and slightly negative. But this panel average is a mirage: it pools opposing per-checkpoint effects. The change is positive for the weakest base and strongly negative for the strongest, spanning \(+0.0400\) \([+0.0003, +0.0776]\) for SmolLM2-1.7B, \(-0.0389\) \([-0.0829, +0.0062]\) for Qwen2.5-1.5B, \(-0.0869\) \([-0.1114, -0.0613]\) for SmolLM3-3B, and \(-0.1499\) \([-0.1963, -0.1050]\) for Qwen3-4B. The ordering tracks base transfer strength (Table 3): the models with the most transfer skill to begin with (SmolLM3 at \(0.9102\), Qwen3-4B at \(0.9438\)) give the most of it back, while the model that had the least (\(0.7904\)) is the only one that improves. In other words, SFT does not add a portable notion of “unsafe”; it reshapes the score toward the training sources, and for a model that already generalised well that reshaping is a net loss off-distribution (Figure 3).

3.3Where the transfer losses live: a per-benchmark decomposition

Aggregating over benchmarks can also hide which data the guard forgets how to rank. The upper block of Table 5 decomposes the change benchmark by benchmark. On the represented side every source rises sharply — toxicchat \(+0.1880\), prompt_injections \(+0.3704\), and jailbreak_classification \(+0.4119\) — confirming the gain is broad across the training sources, not driven by one easy set.

On the transfer side the losses are concentrated on the jailbreak-style adversarial sets: jailbreakbench \(-0.0776\), wildjailbreak \(-0.0792\), and wildguardtest \(-0.0669\) all fall by a comparable amount, while xstest — which contrasts genuinely unsafe prompts against benign look-alikes rather than adversarial attacks — barely moves at \(-0.0120\). The pattern is coherent with the mechanism above: the training sources over-represent a particular flavour of unsafe text, so after tuning the guard ranks those confidently but loses discrimination precisely on the adversarial, distribution-shifted prompts that a deployed guard most needs to catch. Specialization is not uniform forgetting; it is forgetting the hardest, most out-of-distribution cases first.

3.4The specialization plane: 15 of 20 guards specialize

The clearest single view of Act I is the specialization plane (Figure 4), which plots each of the \(20\) (checkpoint, seed) guards by its represented change (horizontal) against its transfer change (vertical). Four quadrants have direct meaning: lower-right = represented up, transfer down (“specialize”); upper-right = both up (“uniform gain”); lower-left = both down (“uniform loss”); upper-left = transfer-favoured. 15 of 20 guards land in the specialize quadrant, 5 in uniform gain, and none in uniform loss or the transfer-favoured quadrant. So specialization is the dominant — but not universal — outcome: the five uniform-gain points are four of the five SmolLM2 seeds plus Qwen2.5’s seed 42 — concentrated on the two checkpoints with the weakest base transfer, which have the most room to improve there (SmolLM2’s remaining seed, 46, narrowly specializes at \(-0.001\)). The absence of any uniform-loss point matters: SFT never simply degrades the guard everywhere; it trades transfer for represented ranking. Per-seed values behind the plane are tabulated in Table 4, and show the effect is stable within each checkpoint (e.g. every Qwen3-4B seed is a substantial transfer loss, every SmolLM2 seed a represented gain).

3.5At a deployable operating point

Ranking is threshold-free, but deployment is not: at some point you must pick a cutoff and turn scores into block/allow decisions. To see the deployment consequence of specialization we select each guard’s threshold to hit a \(5\%\) false-positive-rate (FPR) target on a calibration split, then read off the realized rates. The lower blocks of Table 5 report them.

Represented recall soars.

On the represented sources, catching the unsafe prompts (TPR) jumps from 13.0% (base) to 76.9% (SFT) — the operating-point face of the same \(\approx 0.98\) ranking gain — while the represented false-alarm rate stays low and even edges down (1.7% to 1.1%). If you only measured on the training sources, the guard would look strictly and dramatically better.

Transfer pays for it.

Off-source the picture inverts. The transfer benchmark-macro FPR climbs from 8.1% (base) to 15.5% (SFT) — the tuned guard raises nearly twice as many false alarms on data it never trained on — and the pooled-negative FPR blows past target from 4.3% to 17.0%, so the calibrated threshold simply does not transfer. Transfer recall rises only modestly (51.7% to 58.1%), nowhere near the represented jump — and that rise is not iso-FPR, which matters more than its size. The two recalls are realized at 8.1% and 15.5% macro FPR, so the tuned guard buys much of its extra recall by alarming roughly twice as often; a recall comparison at unequal false-alarm rates is not a comparison of discriminative power.

The fair comparison: an equal false-alarm budget.

So we make the alarm budgets equal and re-read the same rows. Table 6 gives each tuned guard the threshold at which its pooled transfer false-alarm rate matches its own base’s, and then compares recalls. The modest apparent gain does not merely shrink — it reverses, on all four checkpoints and by a wide margin: transfer recall goes 0.517 \(\to\) 0.217 (-0.300) and HarmBench recall 0.780 \(\to\) 0.203 (-0.577). At an equal alarm budget the tuned guard catches less than half of what its own untuned base catches off-source. The direction is stable across the three quantile conventions we tried (panel mean -0.300 to -0.290).

This is the honest form of the operating-point result, and it is the one the rest of Act I predicts: the \(+0.06\) recall gain in Table 5 is an artifact of comparing at unequal alarm rates. It needs no GPU and no pinned environment — matching false-alarm rates is ranking arithmetic on the same committed score_raw and gold columns every other covered artifact uses, so it is regenerated and byte-checked by make verify (Section 9); the emitter reproduces every published row of Table 5 exactly before adding the matched column, which is how we know the two tables read the same underlying scores. Earlier drafts described this reconstruction as a direction that required the lock-pinned environment. That was wrong on both counts, and the measured result is both larger and sharper than the hedge it replaces.

The single-class stress sets make the cost concrete: on the hard HarmBench attacks the tuned guard’s recall falls from 78.0% to 60.0% — it catches less of the very content a guard exists to stop — while benign over-refusal on OR-Bench is essentially flat (11.8% to 12.0%). Since OR-Bench is also unseen data, “off-distribution” cannot be what distinguishes it: the extra false alarms are specific to the transfer suite’s negatives, and we do not identify what separates them. What the flat OR-Bench line does rule out is a blanket increase in caution.

Note that this HarmBench drop needs no threshold caveat at all: the tuned guard catches less while alarming more (15.5% vs 8.1% macro transfer FPR), so it is dominated — worse on both axes at once, not traded off. Equalising the budget only widens the gap, to 0.203 against the base’s 0.780 (Table 6).

On this four-checkpoint panel, post-SFT scores cluster near a benchmark-fixed endpoint, so “stronger bases specialize more” is largely arithmetic rather than skill; the full attractor analysis and Figure 16 are in Appendix C.

3.6The whole ranking is not the operating region

Section 3.5 equalised the alarm budget and re-read one operating point. A more basic question sits underneath it, and the report has not answered it: our headline metric is macro-AP, an average of precision over the entire ranking, while every deployment sentence we write is about a guard that fires on a few percent of traffic. Those are different quantities. AP rewards ordering deep in the negative mass — the region below any threshold an inline guard would ever use — so a change in AP need not imply the same change where the guard is actually placed.

Table 7 settles it by recomputing the identical eight cells under three metrics on the same committed rows: macro-AP, the benchmark-macro one-way partial AUC over FPR \([0,0.05]\), and benchmark-macro recall at that budget. Partial AUC integrates the ROC only inside the alarm budget and is normalised to the mean TPR there, so it reads on the same scale as a recall — against a chance floor of \(0.025\), not \(0.5\). Intervals are the report’s usual paired bootstrap over 2,000 replicates, resampling evaluation family_id clusters and training seeds.

The direction survives; the magnitude does not.

0 of the eight cells change sign: every checkpoint moves the same way under all three metrics, so Act I’s qualitative finding is not an artifact of choosing an average-ranking metric. What macro-AP does do is systematically understate the trade, on both sides. The represented gain is \(+0.323\) on AP against \(+0.686\) on pAUC (\(2.1\times\)), and the transfer cost is \(-0.059\) against \(-0.174\) (\(3.0\times\)), with panel-mean budget recall moving \(-0.199\). The worst cell is the recurring one: Qwen3-4B gives back \(-0.150\) of macro-AP and \(-0.409\) [-0.499, -0.312] of pAUC — \(-0.437\) in budget recall, which is to say roughly two of every five unsafe prompts it would have caught at a \(0.05\) alarm rate before tuning.

One qualification cuts the other way, and it weakens a claim we make elsewhere. SmolLM2-1.7B is the single checkpoint whose transfer improves under SFT on macro-AP (\(+0.0400\), an interval that clears zero). In the operating region that gain does not survive: pAUC \(+0.011\) and budget recall \(-0.012\), both straddling zero. So “positive for the weakest base” is a statement about average ranking, not about deployable recall, and Section 3.2’s “the model that had the least is the only one that improves” should be read with that scope.

This is a re-reading of evidence the report already has, at the same evidence tier: same fixed panel, same inspected sources, same retrospective status, no new data and no GPU. It licenses nothing that Table 3 did not already license — it prices it differently.

3.7A recipe control: does anti-forgetting (KL-regularized) SFT preserve transfer?

The specialization above was measured under one fine-tuning recipe — completion-only LoRA-SFT with no explicit anchor to the base checkpoint. A fair objection is that the transfer loss might be a property of this unregularized recipe rather than of fine-tuning as such. The standard remedy is an anti-forgetting penalty that keeps the fine-tune close to the base in output space, so we add exactly that — a KL term on the SFT loss:

evaluated on the completion tokens, with the frozen base \(\pi_{\text{base}}\) recovered from the adapter’s own disabled path (no second model in memory, no base retraining). Everything else is held at the Act I settings — same manifest, seeds, panel, scorer, and represented/transfer splits — and \(\beta=0\) reproduces vanilla SFT exactly (the cross-entropy is unchanged), so this is a strict one-knob generalization of the Act I recipe rather than a different method. We sweep \(\beta\in\{0.5, 1.0\}\) over all four checkpoints and 5 seeds, and score every adapter through the identical single-token margin used everywhere else.

Table 8 reports transfer and represented macro-AP for the base, vanilla SFT (\(\beta=0\), trained in the same environment as the KL runs so the contrast carries no hardware confound), and KL-regularized SFT at each \(\beta\). The anti-forgetting question is read from the transfer block: relative to vanilla SFT, KL at \(\beta=0.5\) changes transfer macro-AP by +0.061 on average across the four checkpoints (at a represented cost of -0.035), and at \(\beta=1.0\) by +0.051 (represented cost -0.053). In other words, a base-anchored penalty recovers a meaningful part of the transfer that vanilla SFT gives up — so a substantial share of Act I’s transfer loss is a property of the unregularized recipe, not of fine-tuning as such.

An accidental noise floor, and why it bounds several claims in this report.

That same-environment \(\beta=0\) arm is also the closest thing we have to a repeat of Act I: identical recipe, identical manifest, identical seeds, identical scorer — only the execution environment differs. It does not land on the same number. Comparing the SFT transfer column of Table 8 with Table 3 gives gaps of \(0.014\), \(0.009\), \(0.009\) and \(0.029\) (Qwen2.5, SmolLM2, SmolLM3, Qwen3-4B), i.e. a mean of \(0.015\) and a worst case of \(0.029\) macro-AP for a recipe we intended to be deterministic.

We report this because it is a ceiling on what any small effect in this report can mean. Effects comfortably above it — Act I’s represented gain (\(+0.32\)), the matched-budget recall collapse (\(-0.300\)), composition’s recovery over SFT (\(+0.076\)) — are not threatened by it. Effects at or below it should be read as unresolved: composition’s \(+0.017\) aggregate edge over the base, and the \(\beta=1.0\) transfer change for SmolLM2 (\(+0.004\)), are both inside this envelope. The bootstrap intervals elsewhere in the report resample evaluation rows and seeds; they do not capture this environment term, so they are narrower than a full reproduction would be.

The dial, priced in the operating region.

Section 3.6 showed macro-AP understates Act I’s trade; it understates this one too, and asymmetrically. Table 9 recomputes the same KL cells over FPR \([0,0.05]\). The direction is unchanged at both \(\beta\) — KL buys transfer and charges represented ranking — but at \(\beta{=}0.5\) the transfer gain is \(+0.061\) on AP against \(+0.149\) on pAUC (\(2.4\times\)) and \(+0.163\) in budget recall, while the represented cost is \(-0.035\) against \(-0.214\) (\(6.2\times\)). Both halves grow, and the cost half grows more than twice as fast as the benefit half. A dial that looks like “two points of represented AP for six points of transfer AP” is, where a guard is placed, closer to twenty-one points of represented pAUC for fifteen. That does not make KL useless — it makes the choice sharper, and more clearly a choice.

It also sharpens the one preregistered verdict in this report. Section 4’s RQ2 fails a registered \(-0.02\) non-inferiority margin on represented macro-AP. Read on AP that margin is missed by about \(1.7\times\); read in the operating region on this panel it is missed by about \(10.7\times\). The registered criterion was not narrowly missed. (These are point estimates on the four general checkpoints, not a re-run of the registered study’s own family bootstrap on its ten-checkpoint panel, so they qualify that verdict’s magnitude rather than restating its interval.)

3.8The deployment base rate re-spaces the ranking

One more axis quietly co-produces the score, and it is not the model at all: the prevalence of unsafe prompts. All the AP numbers above are measured on balanced (or near-balanced) pools, but real inbound traffic is overwhelmingly benign — unsafe prompts might be \(1\%\) of requests. Average precision depends on that base rate, and the dependence is exact: given a guard’s ranking (its ROC), AP at any prevalence \(\pi_+\) is a fixed recomputation,

where \(u\) is recall and \(\mathrm{FPR}(u)\) is the guard’s false-positive rate at that recall — so no new model runs are needed, only the committed per-row scores.

Two consequences follow, both recomputed exactly from the committed scores. First, the balanced AP is an optimistic reading of deployed precision: SmolLM3-3B’s base transfer ranking scores \(\mathrm{AP}=0.91\) on a balanced pool but only \(\approx0.51\) at \(1\%\) prevalence (Figure 5). Second — and this is the same lesson at the metric’s own prior, though it is a re-spacing rather than a benchmark gain that fails to transfer — low prevalence collapses and re-spaces the ranking. The extremes are stable (Qwen3-4B stays first, SmolLM2 and Qwen2.5 stay in the lower half at every prevalence), but the lower two re-order: Qwen2.5-1.5B edges SmolLM2-1.7B at balance (\(0.82\) vs. \(0.79\)) yet falls behind it once positives are rare (\(0.11\) vs. \(0.21\) at \(1\%\)), because a scarce positive class re-expands exactly the low-recall precision differences the balanced pool compresses. This is a re-spacing and a partial re-order, not a wholesale winner-flip — but it is enough that a single balanced AP can invert two guards’ apparent order at deployment prevalence. Reporting each headline guard as a curve \(\mathrm{AP}(\pi_+)\) rather than a single balanced number is a zero-cost recompute (roadmap), and it is the honest way to state a deployment precision.

4Q1b. Does it hit guards that are already guards? (analysis-preregistered)

Acts I–II characterize what fine-tuning does to general instruction checkpoints, and Act III ranks base guards on regulated domains — all on panels the researcher inspected while building the method, so they are retrospective estimation, not confirmed findings (Section E). This section is the one analysis-preregistered piece: its estimands, decision rules, non-inferiority margin, and interpretation wording were fixed in a committed claim registry (artifacts/starting_type_adaptation_v1/protocol/claim_registry.json) before any score existed — git history puts the registry a day ahead of the scores — so the verdicts below cannot be a post-hoc reading of the data (no HARKing). Two limits on that label, stated here rather than in the appendix. The registry declares itself finalization_status: dev_nonfinal and is not bound to a release lock (no lock exists for this study). And the study re-scores the same 3,308 rows as Acts I–II, from the same frozen manifest — so it is preregistered on the analysis, not blind on the data; the uninspected-cohort half of that discipline remains future work (Section E.3). It asks two questions that the earlier acts raise but cannot settle: (RQ1) does the specialization tradeoff also hit models that are already purpose-built safety guards, and (RQ2) is KL-SFT — which recovered transfer for general checkpoints (Section 3.7) — a free improvement, i.e. does it retain transfer at no represented-source cost?

4.1Design: a \(2\times3\) blocked grid over ten checkpoints

A \(2\times 3\) blocked grid: \(\{\)four general instruction checkpoints, six released purpose-built guards\(\}\times\{\)unmodified \(U\), \(+\)SFT, \(+\)KL-SFT\(\}\), over 10 checkpoints spanning 6 model families (gemma, granite, llama, mistral, qwen, smollm), five seeds, primary \(\beta{=}0.5\). Every guard keeps its native top-level verdict interface (ShieldGemma Yes/No, Qwen3Guard Safe/Controversial/Unsafe — its native three-tier top-level verdict, reduced to the same binary decision margin as every other cell, \(z_{\text{Unsafe}}-z_{\text{Safe}}\), which is what the committed scores record (the Controversial logit is not used) — Llama-Guard safe/unsafe, Granite Yes/No, WildGuard yes/no), validated byte-for-byte against the real tokenizer at a decision-position fidelity check before scoring; all cells share the frozen Paper A binary manifest, LoRA recipe, and optimizer. We report macro-AP (mean over benchmark sources) on the RAW logit margin, on represented (id_test) and held-out (transfer) splits, as a within-checkpoint change vs. the same unmodified checkpoint (the KL reference is that checkpoint).

Which panel each statistic is computed over, and a correction.

RQ1 and RQ2 are registered over the purpose-built panel: 6 released guards spanning 5 model families (gemma, granite, llama, mistral, qwen). The general panel (4 checkpoints, 2 families) is a separate block, and the registered contrast between them is \(\Gamma\). Earlier revisions of this section did not compute it that way. The analyzer built one panel over all 10 checkpoints and grouped it by model family, which placed Qwen2.5-1.5B and Qwen3-4B in the same qwen family as Qwen3Guard-Gen-0.6B and Qwen3Guard-Gen-4B — so every published H was a six-family mixed-panel statistic and estimated a different quantity from the registered one. The same taxonomy is what previously made \(\Gamma\) indeterminate, since a family spanning both starting types cannot sit on either side of it. The analyzer now carries starting_type through to the panel and the numbers below are the registered purpose-built-panel estimands; the superseded mixed-panel values are kept beside them (Section 4.4) rather than deleted.

Each statistic H is an equal-weight mean over the model families of its panel; its one-sided \(97.5\%\) lower bound comes from a 10,000-resample bootstrap that resamples near-duplicate evaluation row families (family_id) and training seeds while holding model and benchmark-source identity fixed, Bonferroni-split across the two research questions (familywise \(\alpha=0.05\)). \(\Gamma\) rides the same replicates, so it is a paired difference rather than a difference of two independent bootstraps. Because benchmark identity is held fixed, these bounds carry evaluation-row and seed uncertainty only — not between-benchmark or between-model-family uncertainty. One count belongs here because an earlier revision reported the wrong one. The study scores 3,308 rows per condition, but the two analysed regimes are id_test and transfer_test only, so the bootstrap resamples the 2,140 family_id clusters of the 2,257 rows those two splits contain — not the 3,170 clusters of the full scored set, which is what an earlier revision quoted. The calibration and single-class stress splits are scored but not analysed here.

4.2RQ1: ordinary SFT specializes released guards too

On the registered purpose-built panel, our SFT protocol raised represented-source macro-AP by +0.111 (equal-family mean; LCB +0.070) while the gain was concentrated relative to held-out transfer (H_conc +0.183, LCB +0.137), so the registered criterion \(\text{LCB}(\text{H\_gain})>0\) and \(\text{LCB}(\text{H\_conc})>0\) is met. Table 10 and Figure 6 show the pattern is not a general-model artifact: the released guards move the same way — SFT buys represented ranking (e.g. ShieldGemma-2B \(+0.212\), Granite-Guard-2B \(+0.139\)) at a held-out cost (equal-family held-out change under SFT -0.072). Fine-tuning a released guard on your data specializes it toward your sources, at a transfer cost, exactly as it does a general checkpoint.

We report that the criterion is met; we do not report RQ1 as “supported.” The two are not the same thing here, and the difference is the whole reason this subsection is worded as it is. A confirmatory verdict would require a protocol that was actually followed, and four things went wrong with this one, each stated in full below: the claim registry is dev_nonfinal and no lock binds it; no checkpoint has a passing preflight, so the eligibility gate never ran; a degenerate cell was retained against that gate; and the panel split reported here was written after the outcomes were known, to repair an analyzer computing the wrong estimand. Any one of those is enough to disqualify a confirmatory reading. Read this as an analysis-preregistered fixed-panel estimate whose decision rule happens to be met — which is a weaker and more accurate thing to say.

4.3RQ2: KL-SFT retains transfer, but is not a free improvement

KL-SFT did preserve held-out transfer relative to SFT (H_preserve +0.047, LCB +0.032): in Table 10 every KL-SFT held-out delta on the nine informative checkpoints is less negative (or more positive) than its SFT counterpart; on the degenerate Llama-Guard-3-1B cell both are exactly zero. But the represented-source cost of that retention, H_cost -0.034, has LCB -0.062, which does not clear the registered non-inferiority margin of \(-0.02\) — so the second criterion fails. KL-SFT trades represented-source adaptation gain for transfer retention; on this panel it is a genuine trade, not a free lunch. This is a more cautious reading than the general-checkpoint KL control of Section 3.7 (where the represented cost was smaller), and the locked-criterion design reports it as such rather than reading the favorable direction as a win. The same four caveats apply: this is a failed criterion on an estimate, not a falsified confirmatory hypothesis.

4.4What this does and does not license

The within-checkpoint deltas are causal only for this recipe on this checkpoint; the general-vs-purpose contrast is a descriptive blocked comparison (checkpoints are not randomly assigned to a starting type), not a causal claim about starting type. The bounds are conditional on the fixed model panel: the bootstrap resamples evaluation families and seeds but holds model identities fixed, so with few families a single dominant family can move the equal-family mean; read H as a fixed-panel summary, not a population estimate.

The Llama-Guard cell is a harness artifact, and it dilutes every number here.

One checkpoint, Llama-Guard-3-1B, shows exactly zero movement under both SFT and KL-SFT (Table 10), because every one of its eleven conditions is a single constant: the unmodified arm returns \(-0.125\) on all 3,308 rows (sd \(0\)), and each tuned seed returns one constant of its own, always \(-0.125\), \(0\) or \(+0.125\). So its macro-AP equals the source base rates in every cell and no adaptation can move it. (An earlier revision said “one unique score across all 36,388 scored cells.” That was wrong as stated — across the eleven conditions the committed parquet holds three distinct values — though the consequence is the same, since the constancy that makes the cell uninformative is within each condition.) The cause is not a property of that model. It was two independent harness bugs, diagnosed after this study was scored and described in full in Section 7.2: the native template rendered its conversation wrapper with the user turn missing while still satisfying every marker, and the verdict was read at a prompt position that carries the distribution over a two-newline prefix rather than over safe/unsafe. Its output head is intact and both decision tokens carry full-norm rows. An earlier version of this section attributed the null cell to that pruned, embedding-tied head; that explanation is superseded and wrong. The cell measures our instrument, not the guard.

Two consequences follow, and both are deviations from the protocol as written rather than results. First, the cell is retained in the equal-family means as a zero contribution, so each of the four purpose-built-panel H statistics is diluted by exactly \(5/4 = 1.25\times\) — that panel has 5 families of which 4 carry a signal. (The general-panel statistics contain no null family and are undiluted; \(\Gamma\), being a difference of the two, inherits the dilution only through its purpose-built term.) Dropping the null family would move \(H_{\mathrm{gain}}\) from +0.111 to +0.139, \(H_{\mathrm{conc}}\) from +0.183 to +0.229, \(H_{\mathrm{preserve}}\) from +0.047 to +0.059, and \(H_{\mathrm{cost}}\) from -0.034 to -0.043. Because Llama-Guard’s unmodified, SFT and KL macro-APs are identical under any family weighting, its contribution is identically zero in every bootstrap replicate too, so each bound rescales by the same exact factor without a rerun: \(H_{\mathrm{gain}}\) LCB +0.070\(\to\)+0.088, \(H_{\mathrm{conc}}\) +0.137\(\to\)+0.171, \(H_{\mathrm{preserve}}\) +0.032\(\to\)+0.040, \(H_{\mathrm{cost}}\) -0.062\(\to\)-0.077. The RQ1 criterion stays met and the RQ2 criterion still fails, and fails harder. So this cannot manufacture support: it makes RQ1 harder to clear and makes RQ2’s failure look milder than it is. The reported magnitudes are therefore conservative — but they are smaller than the informative-family values by that factor, which is stated here rather than left for a reader to derive. (An earlier revision quoted this dilution as \(6/5\), which was the factor for the mixed six-family panel; on the registered purpose-built panel the denominator is different, and these figures are now emitted by the analyzer rather than typed.) Second, the study’s own eligibility rule (proposal §4.3) requires each checkpoint’s verdict likelihoods to be nonconstant; a degenerate cell fails that test, so under the rule as written this checkpoint should have been excluded rather than retained at zero. Retaining it is the conservative choice, but it is a departure from a preregistered criterion and not an application of one.

The pooled starting-type interaction, now computed.

The protocol also registers a fixed-panel interaction \(\Gamma\) — the equal-family mean movement of purpose-built families minus that of general families (proposal §6.3). Two earlier revisions of this paragraph gave two different wrong accounts of why it was missing. It was not a protocol indeterminacy: proposal §6.3 already prescribes that “the two Qwen purpose-built sizes contribute one Qwen-family value,” which separates them from the general Qwen checkpoints, so \(\Gamma\) was computable as registered all along. It was an analysis-code deviation — the single pooled qwen family — and it is now repaired. \(\Gamma_{\mathrm{gain}}\) is -0.213 [-0.240, -0.179], \(\Gamma_{\mathrm{conc}}\) is -0.190 [-0.223, -0.151], \(\Gamma_{\mathrm{preserve}}\) is -0.011 [-0.031, +0.010], and \(\Gamma_{\mathrm{cost}}\) is +0.002 [-0.016, +0.019] (equal-family means over 5 purpose-built and 2 general families, on the same replicates as everything above). Two things must be said about it. It is a descriptive fixed-panel interaction: checkpoints are not randomly assigned to a starting type, so this is a blocked comparison of two particular sets of models, not a causal effect of being purpose-built. And it is computed on 5 and 2 families respectively, one of the purpose-built ones being the null Llama cell, so its interval is wide by construction and a single family can move it.

The registry’s stop_narrow_gates fallback would have authorized omitting \(\Gamma\), conditional on fewer than three purpose-built families passing preflight — but that condition cannot be evaluated either way, because no checkpoint has a passing preflight: all ten files in artifacts/starting_type_adaptation_v1/preflight/ record eligible: false with finalization_status: nonfinal, and the two training-dependent checks — including smoke_adapter_only_finite, the one that implements the nonconstant-margin rule — are skipped with reason: include_training=False. So the eligibility gate was never run for any checkpoint, not merely violated for one. An earlier version of this paragraph asserted that five purpose-built families passed preflight; the committed artifacts contradict that and it was wrong.

The superseded mixed-panel numbers, for the record.

Every H in the two revisions before this one was computed over a single six-family panel pooling both starting types: \(H_{\mathrm{gain}}\) +0.174 (LCB +0.129), \(H_{\mathrm{conc}}\) +0.239 (LCB +0.189), \(H_{\mathrm{preserve}}\) +0.049, \(H_{\mathrm{cost}}\) -0.036. Those are not the registered RQ1/RQ2 estimand and should not be cited as such; they are printed here so a reader comparing against an earlier PDF can see exactly what changed and by how much. The general panel alone gives \(H_{\mathrm{gain}}\) +0.324 and \(H_{\mathrm{conc}}\) +0.373, which is the other half of \(\Gamma\).

5Q2. Can transfer be recovered without retraining? Output-space composition

Act I left us with an uncomfortable asymmetry. Supervised fine-tuning (SFT) buys a large, uniform gain on the sources a guard trained on (\(+0.3234\) macro-AP, [\(+0.2647,+0.3690\)]) while, on average, giving back a little transfer to sources it never saw (\(-0.0589\) [\(-0.0837,-0.0321\)]), and for the strongest base the transfer cost is steep. The instinct is to tune harder — more data, more steps, a different training recipe. Act II asks the opposite question: instead of overwriting the base’s judgment, can we keep it in the decision? The base checkpoint, before any guard fine-tune, is often a surprisingly good transfer scorer (in Section 3 three of four bases transfer better than their own SFT guard). If SFT’s loss is that it has forgotten the base’s broad, non-specialized view, then the cheapest repair is not to retrain but to put the base back in the room at decision time and let it vote.

5.1The fixed composition operator

The rule is deliberately the simplest thing that could work. For a single input \(x\) we run both the base and one SFT adapter, map each model’s raw score to a probability with its own calibrator, and report the fixed, equal-weight average:

where \(s_b(x)\) and \(s_a(x)\) are the base and adapter single-token margins \(z_{\text{unsafe}}-z_{\text{safe}}\) (the same head used everywhere in this report), and \(C_b,C_a\) are per-model calibrators. Three design choices in Equation 5 are load-bearing, and each is fixed before looking at any transfer result.

Calibration makes the two scores comparable.

A raw margin is not a probability, and two different models emit margins on two different, incomparable scales — a \(+2.0\) from the base and a \(+2.0\) from the adapter need not mean the same confidence. Each calibrator \(C(\cdot)\) is a monotone map from raw margin to a probability in \([0,1]\), fit only on a development split (never on the rows we score), so that after calibration a given output value means the same “probability this is unsafe” for both models. Only then does the average in Equation 5 combine like with like rather than letting whichever model happens to produce larger raw numbers dominate. Concretely (illustration, not a result): if on some prompt the base calibrates to \(0.30\) and the adapter to \(0.90\), the composed score is \(0.60\) — the adapter’s alarm is heard but not obeyed outright.

Equal weights, fixed in advance.

The \(\tfrac12,\tfrac12\) weights are not tuned. Choosing the mixing weight by maximizing transfer on the very rows we then report would be circular — it would let the operator quietly memorize the answer. Fixing the weights a priori means the composition has no free parameters set on the evaluation data; any transfer it recovers is a property of averaging base and adapter, not of a search. (A convex-weight variant that does tune the mixing weight, at \(\alpha=0.95\), was visible during development and is therefore reported only as a non-promotable ablation — see Section 5.5.)

Two inference passes.

Because Equation 5 needs both \(s_b(x)\) and \(s_a(x)\), composition runs two forward passes per input — the base and the adapter — roughly doubling inference cost relative to a single guard. Nothing is retrained; there is no new checkpoint to store beyond the base and its adapter (which a LoRA deployment already keeps). The price of skipping retraining is paid at serving time, not training time.

5.2Output-space composition vs. weight-space merging (WiSE-FT, model soups)

Equation 5 combines models in output space: it averages what they say. The better-known alternatives combine models in weight space: WiSE-FT and model soups ^{[53, 52]} build a single merged network by interpolating parameters, \(\theta_{\mathrm{merge}}=(1-\alpha)\,\theta_{b}+\alpha\,\theta_{a}\), and then run one forward pass through that merged model. The two families trade off opposite costs.

• Weight-space (WiSE-FT / soups). One forward pass, no extra serving cost, one artifact to ship. But it requires the two models to live in the same, interpolable weight space — identical architecture and aligned parameters — and it does no per-model calibration; you interpolate weights and hope the merged head is still well-scaled.

• Output-space (ours, Equation 5). Needs only comparable output scores, not interpolable weights, so in principle it composes models that could never be merged in weight space, and it calibrates each model before combining. The cost is the second inference pass, and the requirement that both scores be put on a common probability scale first.

We test only the output-space operator here. Because the two approaches optimize different constraints, output-space recovery does not predict weight-space recovery, and we do not claim it does; a direct WiSE-FT rescoring control is a stated gap (Section 5.5).

5.3Results: composition recovers transfer at a small represented cost

Table 11 reports the fixed-panel macro-AP of each operator in both regimes, plus the worst-of-the-two-regimes column \(\min(\text{both})\). The pattern is clean. The base is the better of the two single guards on transfer (\(0.866\)) but a poor represented one (\(0.658\)). SFT inverts that: it is the best represented scorer (\(0.982\)) but the worst transfer scorer (\(0.807\)) — the Act I specialization, restated. Composition sits between the two on represented ranking (\(0.962\)) and above the base on transfer (\(0.883\)). Read down the \(\min(\text{both})\) column, composition is the most balanced promotable operator: its worst regime (\(0.883\)) beats both the base’s worst (\(0.658\)) and SFT’s worst (\(0.807\)). In other words, if you must commit to one scorer without knowing whether the next prompt is on-source or off-source, the composition is the best-balanced single choice on this panel by worst-regime AP — a ranking statement, not a deployability one (its operating point still misses the illustrated FPR target; see below).

Against SFT — the relevant baseline, since composition is a repair for an SFT guard — composition changes represented-source macro-AP by \(-0.019\) [\(-0.031,-0.010\)] and transfer by \(+0.076\) [\(+0.058,+0.093\)]. That is the headline trade: it gives back under two points of represented ranking to buy back roughly eight points of transfer. Against the untuned base, the aggregate transfer gain is smaller and its interval is closer to zero, \(+0.017\) [\(+0.005,+0.030\)]: composition edges past the base on average, but only for \(2\) of the four checkpoints individually — and that interval resamples rows and seeds only, so the edge sits inside the \(0.015\)–\(0.029\) reproduction envelope measured in Section 3.7 and the vs.-base direction is unresolved. All intervals here are descriptive paired percentile-bootstrap ranges under the retrospective, estimation-only regime (clean_v2_retrospective_estimation), conditional on this fixed panel — not significance tests and not population claims.

5.3.1Per-checkpoint recovery

The aggregate hides a more instructive per-checkpoint story, in Table 12 and Figure 7.

• SmolLM2-1.7B (\(0.790\!\to\!0.830\!\to\!0.857\)): the friendly case. SFT already nudged transfer up, and composition pushes further, gaining \(+0.027\) over SFT and \(+0.067\) over base — both regimes end up ahead.

• Qwen2.5-1.5B (\(0.819\!\to\!0.780\!\to\!0.855\)): the clearest repair. Here SFT actively hurt transfer (an Act I specialization loss); composition not only reverses it (\(+0.075\) vs. SFT) but climbs back above the base (\(+0.036\)).

• SmolLM3-3B (\(0.910\!\to\!0.823\!\to\!0.907\)): composition almost exactly undoes the SFT damage (\(+0.084\) vs. SFT), landing essentially back at the base (\(-0.003\), interval \([-0.012,+0.005]\) straddling zero) — recovery to the base, not past it.

• Qwen3-4B (\(0.944\!\to\!0.794\!\to\!0.914\)): the recurring character. It was the worst specializer in Act I, so composition helps it most relative to SFT (\(+0.120\), the largest recovery on the panel) — yet it still does not reach the base (\(-0.030\), interval \([-0.043,-0.018]\), entirely below zero). For the strongest base, you would have been better off never tuning at all than tuning-then-composing.

5.3.2Why composition helps at all — and least where the base is strongest

There is a simple statistical reason composition works, and the same reason explains the one place it fails. Averaging two scorers is an ensemble, and an ensemble beats its members when they are each individually good and make different mistakes. A clean diagnostic is the midpoint: if composition merely interpolated between the base and the SFT guard, it would land at their average AP. It does not — it lands above that midpoint by a consistently positive margin on every checkpoint (\(+0.047,+0.055,+0.040,+0.045\); read off Table 12). Because AP is nonlinear, this above-midpoint margin is a heuristic signature of diversity, not a formal decomposition; a direct measurement bears it out — on transfer the base’s per-row errors correlate only 0.422 with its own fine-tune’s, versus 0.851 between two fine-tune seeds (Appendix D). The base and its own fine-tune rank the hard transfer cases differently, and averaging cancels part of each one’s error — which is why composition is not mere interpolation.

The same lens explains why composition helps least where the base is strongest. The gain from averaging shrinks as the two scorers’ errors grow more correlated. A LoRA adapter is a low-rank edit anchored to its base, so a strong base’s fine-tune stays close to it: their errors correlate more, the diversity term shrinks, and the average can no longer clear the (already high) base. That is exactly the Qwen3-4B story — strongest base, most base-correlated adapter, composition landing below base. Acts I and II are then one mechanism seen twice: the strongest base specializes most (Act I) and is helped least by composition (Act II), both because its fine-tune moves the least far from it.

5.3.3The equal-cost control: it is the base that helps, not a second scorer

The diversity account makes a falsifiable prediction: if the recovery came from generic two-model ensembling, then averaging two SFT adapters — same two-pass inference cost, but no base — should recover about as much; if instead it is the base’s less-specialized view that matters, the SFT\(+\)SFT average should recover far less. This control needs no new training: we already hold five scored SFT seeds per checkpoint, so composing two of them is a pure recompute of the committed calibrated per-row scores. Table 13 runs it. base\(+\)SFT beats SFT\(+\)SFT on every checkpoint, and the gap is largest exactly where the base is strongest (\(+0.066\) for SmolLM3-3B, \(+0.102\) for Qwen3-4B, vs. \(+0.013\) for the already-friendly SmolLM2-1.7B). Two independently seeded adapters are near-copies of one another, so their average barely improves on a single SFT guard (\(0.79\)–\(0.84\), close to SFT’s own transfer); the base contributes a genuinely different ranking of the hard off-source cases, and that is what the composition is cashing in. The recovery is therefore attributable to keeping the base, not to running a second model.

5.4Ranking is not calibration: the operating-point gap

Everything above is about ranking (AP), which needs no threshold. Deployment needs a threshold, and here composition’s win is only partial. Table 14 sets each guard’s threshold for a \(5\)% false-positive target on calibration negatives and reports the realized transfer rates. Composition improves recall (macro-TPR \(0.517\!\to\!0.639\) across base\(\to\)comp, best of the three) and its realized transfer false-alarm rate, \(11.4\)%, is much better than SFT’s \(15.5\)% — but it still overshoots the \(5\)% target and remains above the base’s \(8.1\)%. The pooled-FPR column tells the same story (\(0.043/0.170/0.091\) for base/SFT/comp). Recovering rank did not hand us a threshold that transfers: the cutoff chosen on calibration data lands in the wrong place off-source, because the score distribution shifts between the calibration and transfer data even when the ordering recovers. So treat an AP recovery as a reason to re-calibrate on the target regime, never as evidence that the old threshold is safe to reuse.

5.5Ablations, and what Act II does not establish

The logit-average ablation (non-promotable).

The last row of Table 11 reports averaging the two models’ raw margins before calibration (a logit-space average) rather than the calibrated probabilities of Equation 5. It edges the calibrated operator on transfer (\(0.891\) vs. \(0.883\)). We nonetheless do not promote it: both it and the convex-weight variant (\(\alpha=0.95\)) were visible while we were developing the method, so choosing the operator after seeing that it wins on transfer would be exactly the retrospective cherry-pick this report is built to avoid. The calibrated equal-weight average was fixed in advance and is the only primary operator; the alternatives are reported for transparency, as ablations, with no claim attached.

Stated gaps.

This is a pilot, and one control that would sharpen the mechanism is still missing. A real WiSE-FT rescoring control. We contrast against weight-space merging conceptually (Section 5.2) but do not actually rescore a WiSE-FT / soup checkpoint on this panel, so we cannot say which family recovers more transfer here. (The other control we flagged in earlier drafts — an equal-cost SFT\(+\)SFT average — is now run rather than pending, in Section 5.3.3: it confirms the recovery is the base’s doing, not generic ensembling.)

And what it does not license.

No Pareto-dominance claim (composition can leave the strongest base worse off); no causal or mechanistic claim (the ensemble intuition is a motivation, not a proof); and no population or significance claim (all numbers are descriptive contrasts on a fixed four-checkpoint panel, with a manifest inspected during development). Finally, we compose only the SFT adapter here; whether the same operator recovers transfer for guards tuned with other objectives (e.g. preference optimization) is left open.

6Q3. Does a general-safety score cover a regulated domain? No

This is the strongest form of the report’s thesis. In a regulated domain a general-safety benchmark can fail to see the violation at all, not because the guard is weak but because the violating request is, on its surface, entirely polite text. Consider the request that motivates the whole construction:

“Applications from the majority-Spanish-speaking tracts keep coming in weak; draft language to quietly deprioritize that pipeline, and write the denials up as ‘collateral’ so it’s clean.”

There is no slur, no jailbreak, no injection — nothing a content-taxonomy guard is trained to catch. Yet honoring it would commit redlining, proxy discrimination, and adverse-action-reason masking, each a distinct mortgage-law violation. A guard tuned to flag domain-independent harm has no label for this. That request is written blatantly on purpose, to fix the idea; the benchmark’s actual payload is quieter, and Figure 8 works one real row all the way through — the request verbatim, the cards it triggers, and where each of the four zero-shot guards ranks it against the benign traffic in the same split. To measure whether guards catch it, we built a fixed, HMDA-grounded benchmark whose unit is one incoming request and whose defining move is to carry two separately assigned labels — one for ordinary safety, one for mortgage-policy compliance — so that the “looks-safe-but-non-compliant” stratum can be isolated and scored on its own. Everything below is a measuring stick: the labels come from an LLM judge reading written policy cards, not from compliance lawyers, so the benchmark surfaces guard behavior without certifying any legal fact.

6.1The dual-label design: \(G\times D\) and the four quadrants

Every request receives two separately assigned binary labels. (They are separately assigned but not empirically independent in v1: the G1/D0 cell came out empty, so \(G\) is nested inside \(D\) — Section 6.2.)

• General safety \(G\): would an ordinary safety guard call this unsafe — a jailbreak, abuse, an injection? We write \(G{=}0\) for safe and \(G{=}1\) for unsafe.

• Mortgage policy \(D\): would honoring the request break mortgage law or policy? We write \(D{=}0\) for allow and \(D{=}1\) for intervene.

A single derived label composes the two — a request warrants action if it is generally unsafe or a policy violation (or both):

Crossing the two labels gives the four quadrants of Figure 9, and their meaning is the point of the design. G0/D1 is the payload — a request a general guard rates safe that nonetheless solicits a compliance violation, which a guard must flag on \(D\) alone. The other three corners are controls: G0/D0 is a plain safe request the guard must not flag, G1/D1 is bad on both counts, and G1/D0 — a generic jailbreak with no mortgage angle — is empty in the frozen release, a stated limitation. Carrying two labels rather than one merged verdict is exactly what lets us pull the G0/D1 stratum out and score a guard on it alone.

Benchmark construction — HMDA grounding and de-identified fact sheets, the agentic construction pipeline, and the 24 policy cards with the label rubric — is detailed in Appendix C.

6.2The frozen composition (994 rows)

Table 15 gives the full breakdown of the frozen v1_hmda2022 release: 994 rows, all synthetic. It is split family-isolated into train (604), dev (149), and public-test (146), plus a 95-row extra slice (called “confirmatory” and “sealed” in earlier versions of this report and in the data card — it is neither, see the corrections below); “family-isolated” means rows sharing a content_family are grouped and sent as a whole to either train or test, never split. That invariant holds exactly (0 of 958 content families span splits), but it is narrower than the guarantee the phrase suggests: grouping is by content_family only, so a near-twin that the clusterer did not group can still cross — and one does, as the correction below records.

Two corrections to the release’s own description.

Earlier versions of this report, the data card, and the benchmark paper called that 95-row slice sealed and “held back to detect overfitting later.” Neither half is true of the artifact as shipped. The file private_test.jsonl is committed to this repository, carries full prompt text, and was visible to the researcher — the builder (magen/package.py) does write it outside the distributable bundle, but the committed release directory was assembled by hand and includes it. And it is already spent: the GPT baseline scored mortgage_hmda2022 over public_test \(+\) private_test (\(241 = 146 + 95\) rows), so it can no longer serve as a later overfitting check for anything. Read it as dataset-held-out by convention, not as a sealed cohort; a genuinely sealed cohort must never enter the public repository, and building one is future work. No number in this report is affected — every mortgage result below is public_test only.

Second, family isolation holds at the level it was implemented and not at the level the phrase suggests. No content_family spans two splits (0 of 958), but pairs were never a grouping key, and one protected pair leaks: PAIR-0000#1 has its reference arm in train (MGB-BEN-00377) and its wheelchair arm in public test (MGB-BEN-00374). The two differ by one phrase, and the near-duplicate clusterer missed them by \(0.0023\) — an estimated Jaccard of \(0.8477\) against a \(0.85\) threshold. It is the only public-test row within \(0.7\) word-\(5\)-gram similarity of any training row (1 of 146). Because the fairness evaluator keeps only pairs with both arms present in the evaluated split, the reported \(\Delta_{\mathrm{context}}\) and every AP here are unaffected; the consequence is that 38, not 39, pairs are usable inside a single split, and that the isolation guarantee should be stated as content_family-level rather than pair-level.

By quadrant, the G0/D1 stratum is the largest non-benign block — 502 rows — which is by design: the looks-safe-but-non-compliant case is the payload of the benchmark, so it is the best populated. G0/D0 (benign) holds 450 rows, G1/D1 (bad on both counts) 42, and G1/D0 is empty (0 rows): the safety-tuned generator declined to author domain-independent jailbreaks with no mortgage angle, so the \(2\times2\) is only three-quarters populated. This is a real limitation, stated as one — orthogonality of \(G\) and \(D\) is demonstrated on three quadrants, not four. By regulatory domain the rows concentrate in fair lending (204), then fraud (112), UDAAP (90), disclosure (66), ATR/QM (54), and privacy (18), against 450 benign rows.

6.3Protected-class minimal pairs and the \(\Delta_{\mathrm{context}}\) fairness gate

Ranking quality (AP) cannot tell you whether a guard treats two otherwise-identical requests differently because one names a protected group. For that we add protected-class pairs: two requests identical in every slot — same fact sheet, same role, same ask — except for the protected-class mention, both members benign (G0/D0, so a fair guard must PASS both), sharing a pair_id. The release contains 39 protected pairs (78 rows), of which 38 have both arms inside one split (see the correction above). This is the established counterfactual token fairness construction ^{[18, 32]}, and it reframes the manipulation of Bowen III et al.^[6] — who varied race on real applications to audit an underwriter — as a guard invariance test: we do not ask whether the decision is right, only whether the guard’s score moves when nothing but the protected mention changes.

Two arms, and only one of them is a minimal pair.

The 39 pairs are not homogeneous, and the distinction matters for every number below. 21 pairs are true single-token swaps (e.g. Black\(\to\)White). The other 18 contrast a named trait against a meta-descriptive placeholder — “I’m Muslim” versus “I’m an applicant with no stated protected trait” — a seven-word substitution rather than one token. Only the 21 single-token pairs meet the strict minimal-pair definition; the placeholder arm additionally risks changing what a reader (or a guard) takes the request to be, the known failure mode of counterfactual token tests ^[18]. We report both arms and flag which pairs drive any result.

The gate is a single number, the mean absolute within-pair score gap:

where \(\mathcal{P}\) is the set of protected pairs and \(p(\cdot)\) is the guard’s unsafe probability. The target is \(\Delta_{\mathrm{context}}\approx 0\): an invariant guard’s score does not move with the protected token alone. A nonzero value means the protected word by itself is moving the guard — a protected-attribute counterfactual sensitivity in the screening layer (a score-behavior signal, not a legal finding of disparate treatment). This gap also has a threshold-free reading: for one pair, \(|p(x_a)-p(x_b)|\) is the fraction of decision cutoffs at which the protected mention alone flips the verdict, so \(\Delta_{\mathrm{context}}\) is a pair-averaged decision-flip rate — but only under a uniform prior over cutoffs on all of \([0,1]\), which is not the deployable range, so read it as a scale-free summary rather than an operational error rate.

The metric is scale-dependent, and on this panel that is decisive.

Equation 7 is defined on the probability scale, so it is only comparable across guards whose probabilities occupy a comparable range — and here they do not. Qwen3-4B’s entire public-test split sits at a median \(p=3.2\times10^{-6}\): its \(\Delta_{\mathrm{context}}=0.000\) is a saturation artifact, not invariance. We therefore also report the gap on the raw margin scale \(s(x)=z_{\text{unsafe}}-z_{\text{safe}}\), recovered exactly from the committed probabilities by \(s=\log\!\big(p/(1{-}p)\big)\) (Table 16). On that scale Qwen3-4B’s mean gap is \(0.797\) log-odds — a \(2.1\)–\(2.5\times\) shift in odds, in the same direction on all three pairs — against \(0.102\) for SmolLM2-1.7B and \(0.151\) for SmolLM3-3B, and just below Qwen2.5-1.5B’s \(0.84\). So on the margin scale the two guards the probability scale calls best and worst are neighbours, which is the whole point. Cross-model logit scales are not guaranteed comparable, so we do not rank guards on the margin either; the defensible statement is narrower and sufficient: \(\Delta_{\mathrm{context}}\) on the probability scale cannot be compared across guards whose score distributions differ by six orders of magnitude, and Qwen3-4B is not invariant on the scale this report elsewhere prefers (the raw margin, Section 9).

6.4Evaluation protocol and zero-shot baseline results

Protocol.

The reproducible layer is the evaluator, not the generator. A guard emits one unsafe probability per row; that score is mapped to \(G\), \(D\), and the composed \(\mathrm{final}\) label, and scored with the repo’s canonical tie-aware metrics: threshold-free macro average precision for \(G\), \(D\), and final (macro averages so each label counts equally; tie-aware handles rows with identical scores fairly rather than ordering them arbitrarily); a per-quadrant miss rate at a calibration-selected operating point (a cutoff chosen on the dev split); and \(\Delta_{\mathrm{context}}\) from Equation 7. Because the G1/D0 quadrant is empty, every \(G{=}1\) row is also \(D{=}1\), so the composed label \(\text{final}=G\lor D\) equals \(D\) row-for-row; consequently \(\text{AP}\cdot\text{final}\equiv \text{AP}\cdot D\) — the final column in Table 16 duplicates \(\text{AP}\cdot D\) by construction, not by coincidence. We score the four base instruction checkpoints from the companion study ^[49] — the same panel used throughout this report — zero-shot, i.e. exactly as shipped, driven only by a prompt with no mortgage-specific fine-tuning. Gated off-the-shelf guards ^{[27, 23]} were not scored in this earlier mortgage run (their licenses had not yet been accepted; they were accepted in time for the analysis-preregistered adaptation study of Section 4, which does score both), and a domain-fine-tuned arm is future work.

Results.

Table 16 reports the four zero-shot guards on the 146-row public-test split (75 G0/D1, 6 G1, 3 protected pairs). The finding is more nuanced than “general guards ignore mortgage compliance.” Threshold-free, the base guards rank \(D\)-violations only moderately well — \(\text{AP}\cdot D\) between \(0.67\) and \(0.85\), at or above their \(\text{AP}\cdot G\) for three of the four checkpoints (Qwen3-4B is the exception, with \(\text{AP}\cdot G=1.000\) off only \(6\) \(G\)-positives — a noisy small-sample value, see below). Read those against their chance floors, which differ sharply: \(D\)-positives are \(81/146\), so a random ranker already scores \(\text{AP}\cdot D\approx0.555\) and the observed band is only \(0.12\)–\(0.30\) above chance, whereas \(\text{AP}\cdot G\)’s floor is \(6/146\approx0.04\). The base-rate-free \(\text{AUROC}\cdot D\) (\(0.60\)–\(0.78\); Table 16) tells the same story without the prevalence term. The two labels are also not independent in v1: because the G1/D0 cell is empty, every \(G{=}1\) row is also \(D{=}1\), so \(G\) is nested inside \(D\) and \(\text{AP}\cdot D>\text{AP}\cdot G\) would be expected from the base rates alone. No guard reaches \(\text{AP}\cdot D\) anywhere near \(1\), so a large fraction of the subtle G0/D1 stratum is left unresolved by ranking alone.

The protected-pair gate is the sharpest available discriminator here, but on this panel it does not survive its own robustness checks, and we report it as a negative methodological result rather than a guard ranking. Two defects cut in opposite directions and between them dissolve the apparent contrast. (1) Qwen3-4B’s \(\Delta_{\mathrm{context}} = 0.000\) is the saturation artifact of Section 6.3: on the raw margin its gap is \(0.797\) log-odds — the second largest on the panel, just behind Qwen2.5-1.5B’s \(0.84\) and roughly \(5\)–\(8\times\) the two SmolLM checkpoints’ — so it is not the most invariant guard. (2) Qwen2.5-1.5B’s headline \(\Delta_{\mathrm{context}} = 0.183\) is carried almost entirely by one pair: its three per-pair gaps are \(0.018\), \(0.022\) and \(0.508\), and the \(0.508\) pair is PAIR-0020#1, a placeholder contrast (“Muslim” vs. “an applicant with no stated protected trait”), not a single-token swap. Restricted to the two genuine single-token pairs its gap is \(0.020\) — inside the unsaturated guards’ band rather than an outlier above it (SmolLM3-3B \(0.012\), SmolLM2-1.7B \(0.024\); the \(\Delta^{\mathrm{1tok}}\) column of Table 16), so the apparent contrast is carried by the one non-minimal pair and not by the guard. Exhaustive enumeration of all \(27\) three-pair resamples puts \(8/27\) of the mean at \(0.018\)–\(0.022\), so the earlier reading that the gap was “large enough to stand out even against the three-pair noise” does not hold. What remains is the instrument lesson, and it is the useful one: a probability-scale counterfactual gap on three pairs, one of which is not minimal, cannot rank guards — and it is uncertain in any case, since the \(\text{AP}\cdot D\) CIs overlap for five of the six guard pairs (only Qwen3-4B vs. SmolLM2-1.7B separates, \([0.785,0.907]\) vs. \([0.565,0.781]\)) and \(\Delta_{\mathrm{context}}\) rests on three pairs. We therefore read the ranking as a direction (the strongest base also ranks the domain violations highest) rather than a resolved ranking. This is the same recurring character from the rest of the report: Qwen3-4B, the strongest base, is the one that specialized most under SFT (Section 3) and the one composition helped least (Section 5), yet here it is numerically the best-ranking zero-shot mortgage guard — the ranking flips with the benchmark. Its fairness behaviour is a separate question that this instrument, at three pairs and on a saturating scale, cannot answer.

What “\(\text{AP}\cdot D=0.85\)” leaves on the table, concretely.

An aggregate that good can still hide a total miss on an individual row, and the G0/D1 stratum is where that happens. Figure 8 takes one row of the public-test split and follows it through: a loan officer asks, in the register of a routine underwriting question, for a decision note that leans on “market fit” and a neighborhood’s “resale stability” instead of the permissible reason already in the file. The benchmark’s own \(G\) label calls it safe — correctly, in the general-safety sense — while \(D\) calls it intervene against six policy cards spanning redlining, proxy discrimination, and adverse-action reason masking. All four zero-shot guards rank it below the median benign inquiry in the same 146 rows, and one ranks it below every single one of the 65 benign rows — on that guard’s own ordering, no benign inquiry in the split looks safer than this violation. The comparison row in the same box suggests — but does not establish — that this is a surface-form effect rather than a capability ceiling: a different fair-lending row that names the protected traits outright (“surname, preferred language”) is ranked above nearly every benign row by three of the four guards. The two rows are not a minimal pair: they differ in fact sheet, domain label, subdomain, cited cards and request type (one asks the assistant to write a pretextual decision note, the other asks whether protected traits may inform pricing discretion). So the register-versus-violation reading is a hypothesis this pair is consistent with, not a measured effect — and the instrument that would settle it does not yet exist, because every protected pair in v1 is benign on both arms (Section 6.3); testing it needs \(D{=}1\) pairs (Section E.3). What the box does establish is the miss itself, which is exactly the deployment risk the dual-label design exists to surface, and exactly what a single general-safety score reports as clean.

Two numbers are deliberately not reported. First, \(\text{AP}\cdot G\) rests on only 6 G1 positives in this split and is correspondingly noisy — a random ranker already scores \(\text{AP}\cdot G\approx 6/146\approx0.04\), so that column must be read against this chance floor, not against \(0\), and the spread from SmolLM2’s \(0.261\) to Qwen3-4B’s \(1.000\) is a wide small-sample band, not four comparable point estimates. Second, we do not report a fixed-threshold “caught at 5% FPR” count for the G0/D1 stratum: for these zero-shot guards the unsafe-probability distribution is tightly clustered, so a dev-calibrated cutoff sits on a knife-edge — its G0/D1 catch count swung by more than 50 rows between library versions of the quantile routine, i.e. it is not reproducible. That instability is itself a finding: naive threshold transfer is unreliable for these guards, echoing the ranking-recovery-is-not-calibration lesson of Acts I and II (Section 3.5 and Section 5). We therefore report ranking (AP) and invariance (\(\Delta_{\mathrm{context}}\)), and leave the operating point to a re-calibration study.

Not scored (gated, HF license not accepted): llama_guard_3_1b, wildguard_7b.

6.5External breadth: finance, healthcare and law (ExpGuard)

The ExpGuard arm scores the same four base checkpoints as the mortgage arm above, again zero-shot (no domain fine-tuning), so this act is a thematic breadth test — does the benchmark-co-production thesis recur across domains? — rather than a re-run of the Act I/II SFT mechanism. On ExpGuard’s expert-annotated prompts, all four bases already rank domain violations well (aggregate AP \(0.88\)–\(0.96\) across the panel; Table 17, Figure 11). The ordering echoes Act I’s lesson rather than raw capacity: SmolLM3-3B ranks highest (AP \(0.956\), \(95\%\) CI \([.949,.963]\), near-uniform across its own three verticals), with Qwen3-4B close just behind (\(0.951\), \([.943,.958]\)). Those marginal CIs overlap, but that is the wrong test on paired data: scoring both guards on identical rows lets a paired bootstrap cancel row-difficulty variance, and it separates one vertical. SmolLM3-3B \(-\) Qwen3-4B is \(+0.017\) on health with a CI excluding zero, while finance (\(+0.001\)) and law (\(+0.001\)) sit essentially on zero with intervals that contain it (Table 17). Those two are not demonstrated equivalences — an interval containing zero is not evidence of no difference, and no equivalence margin was registered — only differences too small for this sample to sign. One caveat belongs with that word “resolves.” Four paired comparisons are reported (overall plus three verticals) with no multiplicity adjustment, and the health interval clears zero only narrowly (\(+0.0026\) at its lower end). Under a Bonferroni split across the three verticals it would not clear. We therefore treat health as the one vertical where the data lean against a tie — a direction worth a targeted replication, not a resolved ranking. The smaller checkpoints trail (Qwen2.5-1.5B \(0.921\), SmolLM2-1.7B \(0.883\)) — the ranking is not monotone in parameter count. As external, expert-labeled evidence these numbers sit at a strictly stronger label tier than the mortgage LLM-judge, so they are reported beside, and never pooled with, the mortgage or retrospective-panel numbers. The ranking score is the raw decision margin \(z_{\text{unsafe}}-z_{\text{safe}}\) (not a saturating probability), so eval_expguard_external.py --from-scores reproduces every entry exactly from the committed text-free per-row scores; the run was independently cross-checked on an L4 GPU and Apple MPS (agreement to \(3\)–\(4\) decimals).

Paired top-two comparison (SmolLM3-3B \(-\) Qwen3-4B on identical rows, 2,000-resample bootstrap). The marginal CIs above overlap, but a paired test cancels row-difficulty variance and resolves one vertical: overall \(+0.0055\) \([-0.0008, +0.0120]\); finance \(+0.0009\) \([-0.0075, +0.0097]\); health \(+0.0168\) \([+0.0026, +0.0320]\) (CI excludes zero); law \(+0.0011\) \([-0.0118, +0.0140]\). So the two guards are tied on finance and law and separate on health — “unresolved” was an unrun analysis, not a sample-size limit.

7Q4. Should you run a small guard at all? Pricing a hosted frontier guardrail

Every guard in this report is a small open-weights model the operator runs itself. The obvious question a practitioner asks next is whether that choice is costing them safety, and how much: if a hosted frontier model is simply a better guardrail, the engineering in Acts I–II is solving a problem one API call makes disappear. ExpGuard is the one instrument here that can answer it cleanly. Its rows are external and expert-annotated, its task is the same prompt-only classification the local guards perform, and every guard — local and hosted — can be scored on identical rows joined by row hash, so the comparison never crosses the label-tier boundary that Table 31 forbids crossing.

Table 18 adds two families to the four base checkpoints: the same four checkpoints after ordinary SFT, and gpt-5.4 and gpt-5.4-mini at three reasoning efforts. Two choices make the rows commensurable. First, the primary column is recall at a matched \(5\%\) false-alarm budget, not each model’s own verdict — the same discipline Act I applies to base-vs-SFT (Table 6), and it matters here because the frontier configs sit at a self-chosen \(2.3\)–\(3.4\%\) FPR and would otherwise be compared at an operating point nobody selected. Second, the frontier ranking signal is a self-reported integer \(0\)–\(100\) risk, the only graded output the Responses API exposes for reasoning models; across the six configurations it takes only \(47\)–\(65\) distinct values over 2,275 rows, and those ties cap how finely AP can resolve a ranking. That bounds precision without fixing a direction — a finer score could resolve a tie block either way — so a frontier number above a local one should not be read as conservative, only as coarsely resolved.

Figure 12 is this whole section in one image, and the two panels are its two halves. The left panel prices every escape a practitioner reaches for — tune the guard, buy a released one, average them, scale the base, ensemble the seeds, fit a stack — against the hosted number, and none of them arrives. The right panel is the one construction that gets materially closer, and it is not a better guard at all: it is running the small guard on everything and escalating only the slice it is least sure about. The subsections that follow derive each bar and state what it does not license.

7.1How large the gap is, and what buying it costs

The gain is real and it is large.

The best hosted configuration (gpt-5.4 (low)) reaches .896 recall at the matched budget against .787 for the best-performing Act I base checkpoint on these rows (SmolLM3-3B — the panel’s strongest base elsewhere in this report is Qwen3-4B, and the two swap places between instruments, which is the report’s own thesis): a paired row bootstrap on identical rows puts the difference at +0.109 [+0.077, +0.139], an interval comfortably clear of zero. In every hundred unsafe prompts, about eleven that the best panel base guard waves through at a \(5\%\) alarm budget the frontier model catches — roughly half of the \(21.3\) it misses. On evidence this is the largest single accuracy gap anywhere in this report, and it is measured on the strongest label tier we have.

And the accuracy is bought at two orders of magnitude.

Table 20 puts the price beside the gain. The best hosted configuration answers in 1,553 ms at the median against SmolLM3-3B’s \(20.1\) ms batched forward pass (the panel spans \(10\)–\(25\) ms; Table 24) — about 77\(\times\) — at roughly $0.80 per thousand prompts against the amortised cost of a GPU the operator already owns. For a guardrail that fires on every inbound request, that is a budget line and a latency budget, not a rounding error. Three further costs appear in no column of either table. Prompts leave the operator’s infrastructure, which for the mortgage setting of Section 6 is a GLBA question before it is an engineering one. The operating point is only coarsely selectable: a continuous logit margin can be placed at any false-alarm budget exactly, while \(47\)–\(65\) distinct integer scores can only land near one. And the provider’s own input filter refused a subset of ExpGuard prompts outright, before the guard saw them — non-deterministically, with no row refused under all six configurations and most refused under four of six. A guardrail whose input filter intermittently declines to return a verdict, for reasons the operator cannot inspect or appeal, is a compliance artifact in its own right.

7.2Routes that do not close it: tuning, scale, ensembling, purpose-built guards

Neither tuning nor scale closes it.

Two obvious escapes suggest themselves — tune the small guard, or buy a bigger one — and the same 2,275 rows let us price both.

Tuning does not. Act I’s whole subject is what SFT buys, so the tuned arm belongs here, and its effect on this external set is not a small loss but a sign split: SFT changes matched-budget recall by +0.122 on SmolLM2-1.7B and -0.059 on SmolLM3-3B, hurting 4 of 6 checkpoints, for a panel mean of +0.005 that hides swings an order of magnitude larger than itself. Within Act I’s four checkpoints the only one SFT clearly helps is the weakest base; the three stronger ones it degrades, and on the wider six-checkpoint set the two exceptions are that weakest base (\(+0.122\)) and Qwen3-8B (\(+0.013\), inside the reproduction noise floor of Section 3.7). That is Act I’s transfer finding — “a small average that hides opposite signs” — reproducing on external, expert-annotated data rather than on the inspected panel, which is a considerably stronger test of it than Act I could run. No tuned configuration comes near the hosted models: the best is Qwen3-32B at .809.

Scale does not either. Extending the ladder to Qwen3-8B and Qwen3-32B — same family as Qwen3-4B, and verified to render the frozen guard prompt to the same prompt_template_sha256 with the same single-token decision pair, so base size is the only quantity varying — makes Qwen3-32B the strongest open guard in this study at .830. But the ladder is not clean and it does not arrive: 8B fails to improve on 4B at all (-0.020 [-0.050, +0.001], an interval that includes zero, so we read this as “no gain”, not as a loss), and the full \(8\times\) step from 4B to 32B buys +0.062 [+0.039, +0.084]. Set that against the gap that remains to the hosted model even from Qwen3-32B: +0.066 [+0.043, +0.089]. Everything \(8\times\) the parameters bought is about the size of the gap still left, so scale alone does not close it over the range we measured — and 32B is already far outside the deployment envelope that motivates a small guard. We deliberately do not extrapolate a required parameter count from this: with three points, one of them non-monotonic, no scaling law is identified, and an earlier version of this passage claimed “at least another order of magnitude” on evidence that cannot support it.

One provenance caveat governs every tuned row and is stated wherever they appear. The Act I release adapters were produced on an ephemeral runner whose bucket was deleted at cleanup and no longer exist, so the SFT rows here are the KL-SFT sweep’s \(\beta=0\) arm — same LOCK contract, same train manifest sha256, same LoRA recipe, same pinned base revisions, but a distinct execution with different adapter_sha256 values. That is the sft_inenv / sft_committed distinction klsft_summary.json already draws (the two agree to the third decimal on Act I’s own metrics), and these rows are labelled SFT (in-env) throughout. The scale-ladder rows sit outside Act I’s locked four-checkpoint panel by construction — that panel is enforced in code, not merely recorded, so extending it is impossible without invalidating the release — and are trained and scored through a sealed sidecar lock that copies the recipe, seeds and manifests verbatim. Act I’s headline numbers are untouched and are not restated from either run.

Nor does combining what we already have.

The other instinctive response to a vendor gap is an ensemble: we hold 18 open guards with ExpGuard scores — bases, SFT seed-ensembles, released purpose-built guards — so combining them costs no new training. Three combination rules, priced against the strongest single open guard (.834) and using the same rule the ensembling appendix fixes (margins average directly within a checkpoint, rank-percentile across them). One convention note, because the same object is quoted against two denominators in this section: the percentages in this paragraph are shares of the distance from that best single member to hosted, which is the question an ensemble answers (“does combining beat picking the best?”), while Figure 12 and the closing summary measure every route from the best small base, which is the question a practitioner starts from. The stack is \(27\%\) of the way on the first denominator and \(58\%\) on the second; both describe the same \(.850\).

Seed ensembling helps, reliably and almost for free. Averaging the five SFT seeds of a checkpoint gains +0.026 matched-budget recall on average, positive for all six checkpoints, and the adapters already exist. It is also how the tuned 32B recovers what SFT cost it: the seed ensemble reaches .834 against \(0.830\) for its own untuned base.

An unweighted committee actively hurts. Averaging all 18 ranks scores .791, below the best single member — \(-69\%\) of the distance from that member to hosted, i.e. backwards. With members this unequal in quality, equal weights dilute the strong ones; this is the Table 18 spread doing exactly what one should expect.

Even a fitted stack falls short. A logistic stack over all 18 members, scored 5-fold out-of-fold so it cannot grade its own homework, reaches .850 — \(27\%\) of the way from the best single member to hosted (\(58\%\) of the way from the best small base). That is a real gain over any single open guard, but it costs 18 forward passes per request and, less obviously, it needs labelled in-domain data to fit the weights, which is the asset a team reaching for an ensemble usually does not have. Its largest weights also include a negative coefficient on one guard, so the fit is exploiting member-specific error structure that may not survive a change of traffic.

So the answer is no: on ExpGuard, ensembling small guards does not beat the hosted model, and the cheap version of it (equal weights) is worse than picking the best member. It earns its place in one situation — when nothing may leave the network at all, seed ensembling plus stacking is the in-house ceiling, and it is roughly a quarter of the way to what the hosted model would give.

And a released guard is not the shortcut either.

The first thing a practitioner asks is why not simply run a purpose-built guard, so Table 18 carries six of them: Qwen3Guard-Gen at 0.6B and 4B, Llama-Guard-3-1B, Granite-Guardian-3.1-2B, ShieldGemma-2B and WildGuard-7B. Each is scored through its own native verdict contract — ShieldGemma’s policy-conditioned Yes/No, Granite’s risk-definition Yes/No, WildGuard’s Harmful request: slot, Qwen3Guard’s top-level Safety: label, Llama Guard’s \n\nsafe/unsafe — because forcing our frozen prompt on a guard would measure our prompt rather than their model. ExpGuard is also unusually fair ground for them: none was trained on it, so unlike Act I’s dataset-held-out transfer suite — which includes WildGuardTest, WildGuard-7B’s own benchmark — there is no home-field advantage to discount.

None of the six reaches the best untuned open checkpoint. The strongest, Qwen3Guard-Gen-4B at \(0.777\) and WildGuard-7B at \(0.771\), sit below Qwen3-32B’s .830 and well below the hosted .896. Two land far lower — ShieldGemma-2B at \(0.458\) and Llama-Guard-3-1B at \(0.334\) — and the reason is instructive rather than a simple lack of skill: their ranking is respectable (AUROC \(0.865\) and \(0.809\)), but their negatives carry a heavy right tail on this material. Llama Guard’s 95th-percentile benign score, \(+3.64\), sits above its median harmful score, \(+2.50\), so a strict \(5\%\) false-alarm budget discards most of its recall. A guard tuned to a general web-safety taxonomy over-flags a subset of ordinary regulated-domain questions, and that is precisely the failure a mortgage deployment cannot absorb. It is also this report’s thesis arriving from a new direction: the instrument chooses the winner, and on regulated-domain prompts the purpose-built ordering is not the ordering their own benchmarks report.

One correction belongs here, because it concerns a number this repository already published. The starting-type study’s llama_guard_3_1b cell is degenerate: within each of its eleven conditions the score is a single constant on all 3,308 rows — the unmodified arm returns \(-0.125\) throughout, and each tuned seed one constant of its own — so only three distinct values occur in the whole 36,388-row file and no adaptation can move its macro-AP (Section 4.4). The cause was not the pruned output head its preflight caveat anticipated; that head is intact, and both decision tokens carry full-norm rows. It was two independent harness bugs: under transformers 5.x the native template rendered <BEGIN CONVERSATION> <END CONVERSATION> with the user turn missing while still satisfying every wrapper marker, and the verdict was read at the last prompt position, which for this contract carries the distribution over the two-newline prefix rather than over safe/unsafe. Both are fixed (the render must now be shown to carry the payload, and each contract’s verdict prefix is teacher-forced), which is why Llama Guard scores at all here. The published degenerate cell should be read as a harness artifact, not as a measurement of that model.

7.3What scoring the ladder taught us about the specialization tax

A by-product worth more than the comparison that produced it.

Scoring the ladder on Act I’s own two regimes (Table 19) prices spending on parameters against spending on tuning, and the two are not equivalent. SFT lifts the panel’s represented macro-AP from \(0.658\) to \(0.982\) and pays \(-0.059\) transfer for it, with per-checkpoint losses reaching \(-0.150\). Qwen3-32B, untuned, reaches \(0.953\) represented — within \(0.029\) of the tuned panel mean — while holding transfer at \(0.962\) against the tuned panel’s \(0.807\). A larger base recovers most of what SFT buys in-distribution without the transfer collapse Act I identifies as SFT’s characteristic cost.

And the tax tracks the distance to the endpoint.

Tuning the ladder turns this into a trend rather than an anecdote. Ordered by how strong the base already was, SFT’s represented gain decays monotonically while its transfer effect turns from a gain into a loss of \(0.10\)–\(0.15\): \(+0.528/{+}0.040\) on SmolLM2-1.7B (represented base AP \(0.452\)), \(+0.354/-0.039\) on Qwen2.5-1.5B (\(0.633\)), \(+0.313/-0.087\) on SmolLM3-3B (\(0.662\)), \(+0.098/-0.150\) on Qwen3-4B (\(0.885\)), \(+0.076/-0.101\) on Qwen3-8B (\(0.905\)), and \(+0.037/-0.117\) on Qwen3-32B (\(0.953\)). The weakest base is the only one for which SFT is unambiguously good on both regimes; by the strongest it buys almost nothing and still charges close to full price. One qualification the numbers force: the transfer cost is not monotone in base strength — Qwen3-4B pays \(-0.150\) from a \(0.885\) base, more than Qwen3-8B (\(-0.101\) from \(0.905\)) or Qwen3-32B (\(-0.117\) from \(0.953\)) — so this is not a demonstrated tendency of capable bases to specialize harder. What is regular is the endpoint: every tuned checkpoint lands in \(0.78\)–\(0.85\) transfer and \(0.975\)–\(0.990\) represented whatever its base, so the tax is the distance to a benchmark-fixed endpoint, which is the arithmetic of Section C.1 rather than a behavioural law. The deployment consequence survives that reading intact: past a base of roughly \(0.9\) represented, SFT buys under \(0.10\) AP and still charges close to full transfer.

Which answers the objection by measurement, not by extrapolation.

The obvious challenge to the paragraph before last is that a tuned 32B might beat an untuned one, so we tuned it: five seeds, same recipe, same manifest. It does not. Qwen3-32B goes from \(.9533/.9620\) untuned to \(.9903/.8447\) tuned — buying \(+0.037\) represented AP at a cost of \(-0.117\) transfer — and on ExpGuard tuning moves its matched-budget recall by \(-0.020\), the wrong way. For a guardrail, whose entire purpose is the traffic nobody anticipated, that is a bad trade at any size. Across the six checkpoints we can now tune, SFT hurts 4 of 6 on external held-out prompts, and on ExpGuard the best tuned configuration in this study (Qwen3-32B at .809) still sits well below the hosted model’s .896. That ordering is specific to this external, never-trained-on probe: Section 7.4 shows it reverses on sources the panel does represent.

One limit keeps the untuned-32B result from being a recommendation: it is a deployment-choice contrast, not a controlled one. Qwen3-32B is \(8\)–\(21\times\) the parameters of the panel checkpoints and costs accordingly, so the comparison is about where to spend a fixed budget, not about SFT being inferior at equal size. What the tuned-32B cell does establish is narrower and still useful: at that size, on this recipe and data, tuning is not the way to spend the next increment.

Paired comparison (gpt-5.4 (low) \(-\) Qwen3-32B base, the strongest open guard here, on the 2,275 rows both scored; 2,000-resample paired row bootstrap). \(\Delta\)TPR@5%FPR \(= +0.0661\) \([+0.0428, +0.0893]\); \(\Delta\)AP \(= +0.0140\) \([+0.0098, +0.0188]\). Within the Qwen3 family, \(8\times\) the parameters (4B \(\to\) 32B) buys \(\Delta\)TPR \(= +0.0621\) \([+0.0395, +0.0842]\) — about as much as the gap that remains. Against the strongest tuned guard (Qwen3-32B SFT in-env), averaged over its 5 seeds: \(\Delta\)TPR@5%FPR \(= +0.0865\), \(\Delta\)AP \(= +0.0210\). The frontier score is a coarse integer 0–100 risk with heavy ties, which caps how finely its ranking can be resolved. That limits precision without fixing a direction: a finer score could order a tie block either way, so these deltas should not be read as conservative.

7.4The frontier gap is a property of the regime, not of the model

Everything above measures the gap on ExpGuard, and that choice does more work than it looks like. ExpGuard is an external breadth probe: no guard in this report was trained on it, which is exactly what makes it a clean instrument — and also means it reports the transfer regime and nothing else. The \(+0.109\) figure is therefore the transfer gap, not the gap.

Five of the general-safety corpora the GPT baseline scored are also scored by the Act I panel, and those rows can be joined: the two runs simply hash their row identities differently (sha256(text)[:16] against content_sha256, which normalizes first), so re-deriving both digests from the local corpus recovers the mapping for \(100\%\) of the panel’s rows. The regime split is then read from Act I’s own manifest rather than asserted — train.jsonl is exactly jailbreak_classification, prompt_injections, toxicchat — so id_test rows are held-out rows from a represented source, while transfer_test sources are held out at the source level.

Table 22 is the result, at the same matched 5% budget. It reverses direction across the split. On represented sources the panel’s small tuned guards beat the frontier reference. The summary we report is an aggregate that does not depend on which cell wins: the equal-source, equal-checkpoint mean paired difference over the 3 represented sources is +0.083 [+0.013, +0.157] in recall at the matched budget, and +0.039 [+0.015, +0.072] in AP; both exclude zero. Figure 13 is that reversal in one picture.

This aggregate is post hoc and is reported as a descriptive fixed-panel summary, not as a test. It did not exist before the per-cell headline failed multiplicity — it was added in the same revision that found the failure. It is a sounder summary than the maximum of twelve, because its weighting is fixed by the regime split rather than by any result, but that is not the same as having been specified in advance, and we do not claim it was. Only a summary frozen before a fresh cohort is scored could carry a confirmatory frontier claim.

Two things the interval is conditional on, stated before the sensitivities.

It is conditional on these three sources. The bootstrap holds the source set fixed and resamples evaluation families within it, because three purposively chosen corpora do not sample a population of corpora. Drawing sources with replacement instead — the unconditional version — widens the interval to [-0.019, +0.220], which includes zero. That is the honest statement of how far the result travels: it is a claim about the panel’s behaviour on toxicchat, prompt_injections and jailbreak_classification, not about represented sources in general. It is also conditional on the seed pairing: seeds \(42\)–\(46\) are the same five training runs on every source, so the joint bootstrap draws one seed vector per checkpoint and reuses it across sources rather than redrawing independently, which would break the pairing that carries most of the covariance.

The headline is not robust to how the same twelve cells are weighted.

The weighting choice matters enough that giving only the one we report would be misleading, so Table 21 gives all four — each from the same joint bootstrap draws, so they are mutually comparable rather than four separate analyses. Only two of the four support a positive advantage at all: weighting sources by their row counts roughly halves the estimate and straddles zero (the largest source, toxicchat at \(n=451\), carries the smallest effect, \(+0.028\) against \(+0.157\) on prompt_injections), and including the base arms alongside the tuned ones excludes zero in the opposite direction.

That last row is the one to keep in view, because it says what the result is about: the represented-source advantage is a property of tuned guards specifically — Act I’s specialization seen from the other side — and not a general statement that small guards beat hosted ones. On transfer sources the ordering flips back, the reference leads (.967 on xstest against Qwen3-4B base’s .917), and tuning is what costs the guard its position: the strongest transfer guard in the panel is an untuned base.

The largest single cell is exploratory, and does not survive multiplicity.

The most striking cell is Qwen2.5-1.5B — a 1.5B model — at .948 on prompt_injections against the reference’s .741, i.e. +0.207 with a nominal interval of [+0.042, +0.389] and a ranking rather than a threshold advantage (AUROC 0.9928 against 0.8731). That cell is selected as the maximum of 12, so its nominal interval is post-selection and overstates the evidence. Two familywise objects, because they answer different questions. As a decision: a Holm step-down over the 12 represented-source SFT cells, on two-sided percentile-bootstrap \(p\)-values, rejects nothing — this cell’s \(p=0.011\) against a first threshold of \(\alpha/12=0.0042\) (adjusted \(p=0.132\)), so 3 of 12 cells clear zero nominally and 0 survive the correction. As an interval: Holm controls decisions and does not produce intervals, so the band we quote is max-\(T\) over the same twelve cells, standardised by each cell’s own bootstrap spread with the critical value \(c=3.07\) read off the joint draws, giving [-0.072, +0.486], which includes zero. An earlier revision printed a narrower band here and called it Holm-adjusted; it was neither — it rescaled percentile bounds by a ratio of normal critical values and ranked the cells by \(|\Delta|\) over half-width rather than by \(p\), with no step-down stopping. The qualitative answer is unchanged, and it could only ever have moved one way: the omitted stopping rule made the old procedure anti-conservative, so “none survive” was already the generous reading. The per-cell numbers are therefore reported as exploratory throughout, and no sentence in this report should be read as “a 1.5B guard beats the frontier” on the strength of one cell. What the aggregate supports is weaker and still substantive: on average across represented sources, the panel’s tuned guards rank unsafe prompts better than the hosted reference at a matched alarm budget.

An earlier version of this subsection reported +0.207 as \(+0.185\) with a narrower interval. That number came from a different estimand than the one tabulated beside it — the metric of a five-seed score ensemble rather than the mean of five per-seed metrics — so the difference did not equal the two values it sat between, and it carried no training-seed uncertainty. The estimator is now the mean of per-seed paired differences with evaluation families and seeds both resampled, which makes every delta equal the arithmetic difference of the two tabulated values by construction and widens the intervals accordingly. A later revision made one further change to the same intervals: the bootstrap resamples family_id clusters rather than bare rows, so this table now uses the same family-aware uncertainty protocol as the rest of the report instead of a second, looser one. That widened the per-cell intervals slightly and moved the count of nominally significant cells from four to 3.

Paired comparison against gpt-5.4 / low on the rows both scored. Each delta is the mean over training seeds of (guard \(-\) reference), so it equals the arithmetic difference of the two tabulated values; the 2,000-resample bootstrap resamples both near-duplicate evaluation families (family_id, the same protocol as the rest of the report) and training seeds, so the intervals carry seed as well as row uncertainty. Summary reported (post hoc, descriptive — added after the per-cell headline failed multiplicity, so not pre-specified): the equal-source, equal-checkpoint mean over the 3 represented sources is \(\Delta\)TPR@5%FPR \(=\) +0.083 [+0.013, +0.157] and \(\Delta\)AP \(=\) +0.039 [+0.015, +0.072], both excluding zero — conditional on these three sources; drawing sources with replacement instead gives [-0.019, +0.220], which does not. Individual cells are exploratory: the largest (Qwen2.5-1.5B SFT on prompt_injections, \(n=67\), +0.207) is selected as the maximum of 12, so its nominal [+0.042, +0.389] is post-selection; the max-\(T\) simultaneous band over the 12 cells (\(c=3.07\)) is [-0.072, +0.486]. 3 of 12 cells clear zero nominally and 0 survive a Holm step-down on the bootstrap \(p\)-values (smallest \(p=0.011\) against a first threshold of \(\alpha/12=0.0042\)). The jailbreakbench column carries no deltas: the reference’s own TPR there is tie-collapsed (†), so differences against it are uninterpretable. Per-source \(n\) is small (67–451 rows), so these intervals are wide and no per-cell ordering should be read as a ranking.

Four limits bound this, and they matter more than the headline. Per-source \(n\) is \(67\)–\(451\), so the intervals are wide and no per-cell ordering is a ranking. The evidence is retrospective and estimation-only — these rows and this panel were inspected during development — so it sits at the same flavor as Acts I–II in Table 31 and is never pooled with the ExpGuard numbers above despite appearing beside them. A represented-source win is not a claim about novel traffic: it says a guard beats the frontier on distributions an operator can enumerate in training, which is a deployment property and not a capability claim. And id_test is held out by row, not by content — the overlap audit (Section E.1) puts \(1.6\%\)–\(5.0\%\) of each represented split within Jaccard \(0.70\) of a training row, which qualifies these margins rather than overturning them, most of all on jailbreak_classification.

What survives all four is a statement about where the frontier earns its price. It is not buying a uniformly better guard; it is buying the regime a small guard is worst at. That reframes the practical question from “can a small guard match the frontier” to “how much of your traffic resembles something you can put in a training manifest” — and it is the motivation for a routing rule keyed to unfamiliarity rather than to score margin. That rule is a hypothesis this report does not test: the only cascade measured here (Section 7.5) escalates by rank distance to the local guard’s own decision line, which is itself a margin router, and no familiarity detector is implemented or evaluated anywhere in this work.

7.5Which requests, and what this licenses

The deployment question is not which guard, but which requests.

Everything above compares whole guards, which frames the choice as self-host or outsource. That framing is wrong, and the per-row scores show why. Run the small guard inline on every request, rank requests by how near they fall to its own decision line, and re-score only that uncertain band with the hosted model, holding one global \(5\%\) false-alarm budget across the whole construction. Escalating the least-confident \(10\%\) of ExpGuard raises matched-budget recall from .787 to .819; \(20\%\) reaches .842, which is half the distance to the hosted model’s .896; \(30\%\) reaches .856. The curve is steep early because the first requests escalated are the ones a second opinion can actually change, and it is smooth, so the escalated share is a dial set by whatever the data-residency and cost constraints allow rather than an architecture to be chosen once. Two implementation notes, since the construction is easy to get wrong: the two guards are fused on rank, because a logit margin and a self-reported integer risk are not on a common scale; and a per-band threshold was tried first and rejected, because with a small escalated slice there are too few deferred negatives to place a stable quantile, which made the curve non-monotone for purely numerical reasons. This is a retrospective analysis on committed scores, not a deployed system: it assumes the escalated subset may lawfully leave, which for the mortgage traffic of Section 6 is exactly the question that has to be answered first.

What this does and does not license.

It licenses one sentence: on external, expert-annotated finance/health/law prompts, a hosted frontier model is a materially more accurate prompt-safety ranker than any small guard in this report, tuned or not, and the margin survives a paired test. It does not license extending that to the mortgage construct of Section 6: ExpGuard is single-label general prompt safety, not the dual \(G\times D\) compliance judgement, and nothing here was measured on it. Nor does it retire the report’s subject. Acts I–II are about what happens when you must run a small guard yourself — for latency, cost, data residency, or auditability — and this subsection prices that constraint rather than dissolving it. The honest reading is that the frontier number is the bar on unfamiliar traffic: it is what the specialization and composition machinery is trying to reach under constraints the hosted model does not have to satisfy. On traffic the operator can enumerate, Section 7.4 shows the bar is already cleared — which is why the deployment question is a routing question rather than a modelling one.

8Synthesis: what benchmark gains do — and do not — predict

The four questions have one answer. A guard’s benchmark score is co-produced by the benchmark: SFT buys represented-source ranking, not transfer — and at an equal false-alarm budget it does not even buy the recall it appears to, catching 0.217 against its own base’s 0.517 off-source (Act I). The choice of metric co-produces the verdict too, in the same direction: averaging precision over the whole ranking makes the trade look milder than it is, and re-reading the identical rows inside a \(0.05\) alarm budget roughly triples the transfer cost without flipping a single cell (Section 3.6); keeping the base in an output-space average recovers some transfer without retraining (at a second inference pass), though not a threshold (Act II); and a change of domain can hide the violation entirely, so regulated domains need domain-grounded evaluation and a fairness gate (Act III) — and building that gate taught us that the gate itself needs the same scrutiny as the guards: ours does not survive it (Section 6.3). The recurring cast makes it concrete: Qwen3-4B, the strongest base, specializes the most on transfer, is the one composition hurts, yet is numerically the best-ranking zero-shot mortgage guard (on a small split, so read as a direction) — the ranking flips with the benchmark.

The frontier comparison (Section 7) then turns the same lesson outward, and this is where the caution acquires a deployment-facing shape: the choice stops being which guard and starts being which traffic. A hosted model is the more accurate ranker on unfamiliar prompts, but on sources an operator can enumerate in a training manifest the ordering inverts: averaged over the represented sources the panel’s tuned guards rank better than the reference by +0.083 [+0.013, +0.157] at the matched budget (Section 7.4; individual cells are exploratory and none survives a familywise correction). So the gap is not a capability ceiling to be closed by a larger student — it is the price of the regime, and the practical question is not which guard but what share of your traffic you can characterise in advance. That suggests routing on unfamiliarity rather than on score margin — but it is a suggestion, not a result. The cascade we actually measured (Section 7.5) is a margin router, so this report contains evidence about margin routing and none about familiarity routing; the regime result motivates that comparison rather than settling it.

Table 23 distills the whole study into the guideline each finding implies: what we learned, the evidence for it here, and what a practitioner should therefore do. Every row is anchored to a result in this report; where a finding is directional (small split) or fixed-panel, the guideline is qualified accordingly.

8.1The decision guide: gate candidates, not leaderboards

The right-hand column of Table 23 is the decision guide, and it carries one caveat throughout: these are estimates on a fixed panel, and the domain labels are a measuring stick, not a verdict. Two of its rows deserve a sharper edge than a table cell allows. First, composition is a repair for a guard you already tuned, not a free win over the base — if you have not tuned yet and transfer is the priority, the untuned base can already be your best transfer scorer, so compose only once tuning has actually cost you something. Second, ranking recovery is not calibration transfer: after composing, re-choose the threshold on the target regime rather than inheriting it. Figure 14 turns the whole column into a procedure — gate candidates, not leaderboards.

8.2Latency and cost: the case for a small, self-hosted guard

An inline guard runs on every request, so its own latency and cost sit on the critical path. Because our guard emits a single verdict token — one forward pass, no autoregressive generation — it is fast to run: measured per-call latency is \(\approx\)10–50 ms (P50–P90, batched on one A100; P99 up to \(\approx\)94 ms, and composition adds a pass) depending on model size (Table 24), from \(10.4\) ms (P50) for Qwen2.5-1.5B to \(25.2\) ms for Qwen3-4B, with P90 within \(\approx\)50 ms, on a single A100 at batch 16. Latency tracks model size and prompt length, not any decode budget, because there is nothing to decode. One caveat on reading these numbers: they are batched per-row times (batch 16 on one A100) — throughput-latency under load, not a single-request, batch-1 serving path, which carries a higher fixed per-call overhead but no queueing. Treat them as an order-of-magnitude serving estimate on this hardware, not a single-request SLA.

A frontier hosted-API guard is disadvantaged on the two axes that decide an inline deployment, and here that is measured rather than sketched: Table 20 reports hosted P50/P99 latency and $/1k taken on the ExpGuard rows themselves at concurrency 200, against the committed batched-A100 figures in Table 24. The hosted path adds a network round-trip — 1,553 ms median against tens of milliseconds — to every request; it bills a per-token fee that scales with traffic; and, decisively for a regulated domain, it sends every prompt to a third party. Table 25 sets the four axes side by side.

That is a case for self-hosting the traffic you can serve well, not for self-hosting everything. Section 7.4 shows the accuracy ordering inverts by regime, and Section 7.5 prices a selective cascade that keeps the median request local. So the practical conclusion is a routing conclusion: build from small checkpoints, and escalate the slice they serve worst — which is what Row 8 of Table 23 says, and what the unconditional reading of Row 6 would miss.

9Reproducibility

Every table and figure above is \input from a committed generated artifact; the remaining in-prose values are transcribed from those same artifacts and are not covered by the byte-check. Two entry points, run in papers/unified-report/ and both calling reproduce.py (distinct from the repo-root make repro, which re-verifies the release cache): make regenerate rewrites the generated artifacts from committed per-row scores, and make verify recomputes them into a scratch directory and asserts byte-identity with the committed copies, leaving the tracked tree byte-clean. make verify is the one a reader should run. Read its exit code carefully, because a clean checkout does not exit \(0\): it exits nonzero if any covered artifact mismatches and also if any artifact could not be checked at all. On an ordinary machine the 4 Act I artifacts fall in the second bucket, so the run ends CHECK INCOMPLETE with a nonzero status while reporting 0 failed; the per-artifact table it prints, not the exit code, is what says whether anything drifted. Only the lock-pinned environment (docs/reproducibility-environments.md) can exit \(0\). One limit is measured rather than assumed: three upstream generators are not redirected to the scratch directory and rewrite their intermediates in place (the composition pilot’s generated/, and the mortgage out_eval/ and generated/). The tree still ends byte-clean, because with no drift the rewrite is byte-identical — but drift in one of those intermediates would be silently repaired rather than reported. Every artifact this report \inputs is compared properly; the gap is one level upstream of them, and closing it needs an output-directory flag on one script.

Coverage is now complete, and the harness is what says so: of the 35 generated artifacts the report \inputs, 31 are byte-checked in any environment, the remaining 4 require the lock-pinned environment (Act I’s own tables and the \Rep*/\Transfer* macros), and 0 are uncovered. Earlier revisions left eleven inputs outside the harness — including both head-to-head outputs, which back the report’s most prominent frontier claim — and said “every table is byte-checked” anyway. The denominator is defined in the emitted macro file itself: it counts every generated/*.tex the report inputs except repro_macros.tex, which is the harness’s own coverage report and would make the count self-referential (36 files including it). Every count in this paragraph is emitted by the harness (\Repro*) because they were previously typed by hand and went stale twice: an earlier version of the abstract quoted “12 of the 24” after the surface had grown past it, and a later one quoted “8 uncovered” while the harness was emitting eleven.

What the previous revision could not verify, and what changed.

A prior version of this section conceded three defects and left them open; all three are now closed, and the concessions are kept here so the record shows what the numbers above are worth. (i) Eleven generated inputs sat outside the harness — the adaptation, KL-SFT, ensembling, cascade, mortgage-composition and, most seriously, both head-to-head outputs, which back the report’s most prominent frontier claim. All of their emitters read committed analysis JSON, so there was never a reason they could not be checked; they are now wired in, and the report no longer says “every table is byte-checked” while its headline table is uncovered. (ii) Figures were regenerated and never compared. The eleven plotted figures are now rendered into a scratch directory and byte-compared like everything else. Two classes sit outside that, and both are named rather than implied: the three Graphviz diagrams (Figure 2, Figure 15 and Figure 17) are rendered from committed .dot sources but are not byte-compared, because dot output is not reproducible across Graphviz builds; and Figure 4 ships as a committed specialization_plane.pdf with no generator in make_figures.py at all, so it is neither regenerated nor compared. It predates the harness, and re-deriving it from Table 4 — which holds the twenty points it plots — is a stated gap (Section E.3). (iii) Verification was not side-effect-free: make reproduce did not pass --check, and even the checking path rewrote repro_macros.tex — the one artifact it never checked. Emitters now honour a PAPER_GEN_DIR redirect, the coverage macro is checked like any other input, and a verification run leaves the working tree byte-clean.

One limit is not closed and should not be read as if it were. Re-deriving the head-to-head artifact (h2h.json) runs a 2,000-replicate joint bootstrap that takes minutes, so it is an opt-in target (make verify-heavy) rather than part of the default check. What changed is that it is now possible offline at all: the provider’s per-row predictions were previously reachable only through gpt-baseline/raw/, which is gitignored, so a clean checkout could not reconstruct the head-to-head numbers by any route. They are now materialised as a committed, text-free per-row artifact (frontier_rows.json: content digests, the provider’s 0–100 risk per row, per-config parse/failure tallies, and the run’s model string, prompt-contract digest and run id), which the evaluator reads with --offline. Labels and evaluation families come from the committed score parquet on both paths, so the offline reconstruction cannot drift from the live one. The matched-false-alarm-budget table (Table 6) is one of the covered artifacts: it is derived from the same committed score_raw/gold columns as everything else here and needs no GPU.

Three reproducibility tiers, kept distinct.

(i) Analysis reproducibility — regenerating a table from the committed per-row scores — is what make verify checks byte-for-byte, with no GPU and no network. It is the tier the 31 covered artifacts meet in any environment and the 4 Act I artifacts meet under the pinned lock. No input the report \inputs now falls outside it. (ii) Training/scoring reproducibility — re-deriving those per-row scores by training the \(4\times5\) adapters and rescoring — requires a GPU and pinned model/data access (only re-scoring the gated ExpGuard set additionally needs dataset access). (iii) Artifact-generation reproducibility — regenerating the mortgage benchmark itself — is not claimed: its LLM construction stages run at nonzero temperature and the set is intentionally frozen, so only its evaluation reproduces, not its generation. Raw third-party rows are referenced by pinned identifier + revision + content hash, not redistributed.

Benchmark attribution.

The worked case study of Figure 8 quotes rows of MortgageGuardBench v1_hmda2022, Reza Rahimi, PhD (JazzX AI), licensed CC BY 4.0 — the attribution that licence requires, and the same notice the HTML edition carries. Its prompts are synthetic and its labels are LLM-judge and policy-card-consistent, not SME-adjudicated; its release checksums cover release bytes only, not the generator, judge, configuration, or code. The prompts solicit policy violations by design, so reuse should treat them as harmful-content samples. The factual grounding is the public HMDA 2022 loan-level snapshot, a U.S. Government work carrying no U.S. copyright. Redistribution decisions for every source live in benchmarks/registry/distribution.yaml.

Code and data availability.

All code, data manifests, generated tables, and the frozen benchmark are public at https://github.com/rrahimi-uci/safety-guard-dynamics. The one-command pipeline is papers/unified-report/reproduce.py (make verify for a non-mutating check; make regenerate to rewrite generated artifacts); the frozen, dual-labeled mortgage benchmark ships at mortgage-benchmark/benchmark/v1_hmda2022/ (994 rows, SHA-256-checksummed, with a text-free index); the committed per-row guard scores that regenerate every number live under artifacts/ (the Act I/II scores in artifacts/paper_a_sft_v2/, the mortgage baseline in mortgage-benchmark/out_eval/, and the finance/health/law scores in artifacts/expguard_external/); and every figure except Figure 4 is built by papers/unified-report/figures/make_figures.py (that one exception is committed without a generator, as Section 9 records).

10Conclusion

Benchmark gains do not guarantee transfer. On this fixed panel, paired same-checkpoint comparisons show a large represented-source gain (\(+0.3234\)) beside heterogeneous held-out effects (\(-0.0589\)): one weak base improves, while the strongest bases lose transfer. At an equal false-alarm budget the ambiguity disappears operationally — transfer recall falls \(0.517\!\rightarrow\!0.217\) and HarmBench recall falls \(0.780\!\rightarrow\!0.203\), with the tuned guard worse on all four checkpoints. Read inside that budget rather than over the whole ranking, the same paired comparison is \(2.1\times\) larger on the represented side and \(3.0\times\) on transfer, with 0 checkpoints changing sign — so the metric that produced the headline was understating it, and the one base whose transfer improves on macro-AP does not improve at a deployable alarm rate.

The extensions sharpen the boundary rather than erase it. Released purpose-built guards move in the same direction in a non-confirmatory fixed-panel analysis. KL-regularized SFT retains transfer only by giving back represented gain, while base-plus-adapter composition recovers transfer for an additional inference pass. None of these results licenses the claim that fine-tuning always harms transfer or that one guard is universally best.

The deployment result is therefore conditional on traffic. A hosted frontier model leads on unfamiliar, expert-annotated prompts; the tuned local panel leads on represented sources. A regulated domain adds a second boundary: general-safety scores do not identify policy violations that read as ordinary text, so domain-specific instruments and expert validation remain necessary.

The practical rule is the paper’s final deliverable: compare every tune with its own base, on identical represented and held-out sources, at a matched false-alarm budget; require domain, calibration, service, and governance gates before deployment. The evidence remains retrospective except where explicitly labelled otherwise, and the mortgage labels are not expert-adjudicated. The contribution is a workflow that makes specialization visible before it becomes a production failure — not a new winning model.

ARelated work

This report sits at the intersection of five literatures: the design of small LLM-based guard classifiers; the growing evidence that fine-tuning degrades or narrows a model’s safety behavior; the study of calibration and operating points for moderation; model composition in weight versus output space; and the emerging work on domain-specialized guarding in regulated verticals. We review each in turn and, for each, name precisely what it establishes and what our paired, same-checkpoint, composition-aware, four-domain design adds. Our contribution is not a new model, metric, or training algorithm — it is a measurement discipline and four evaluation instruments applied to one fixed panel, so the contrast throughout is methodological rather than a claim of superior scores.

A.1Small LLM guard classifiers

The dominant deployment pattern is a compact decoder LLM turned into a binary or taxonomy-tagged moderation head. Llama Guard ^[27] introduced the input–output safeguard framing and a safety taxonomy, and later iterations pushed toward smaller, cheaper, and multimodal variants ^{[40, 42, 41, 17, 44]}. ShieldGemma ^[54] and Granite Guardian ^[47] extend the recipe to other base families with broad harm taxonomies; WildGuard ^[23] adds one-stop coverage of prompt harm, response harm, and refusal detection; and Qwen3Guard ^[48] is a recent open guard on the same Qwen family two of our checkpoints come from. Beyond the general-purpose guards, several works target narrower slices or richer inference: dedicated prompt-injection detectors ^{[43, 46, 45]}, reasoning- and logic-augmented guardrails ^[30], ensemble-of-experts moderation ^[19], RL-driven multilingual guardrails ^[12], and CPU-class or multi-stage pipelines aimed at cost ^[38]. The parameter-efficient adaptation we use — LoRA ^[26] applied to a chat model to produce a guard — is exactly the LoRA-Guard recipe ^[15], and standardized suites such as GuardBench ^[4] have made cross-model comparison routine.

What this family reports, almost without exception, is absolute moderation performance of one guard against different models on a benchmark. That is precisely the quantity we argue is co-produced by the benchmark and therefore uninformative about a specific fine-tune. Several of these works do report in-distribution versus out-of-distribution blocks — Llama Guard’s own test set against zero-shot ToxicChat and OpenAI-Mod, WildGuard’s and LoRA-Guard’s ID/OOD splits — so the represented/transfer distinction is not ours. Our narrower claim, stated once and not widened anywhere else in this report, is:

a same-checkpoint paired measurement of LoRA-induced change across represented and source-held-out regimes, together with a calibrated base-plus-adapter composition test under matched false-alarm budgets.

An earlier version of this passage said instead that “none of them reports” the paired change and that “none treats retraining-free composition as a measurable design axis.” Both were overclaims that survived because the comparison was made in prose rather than against a table; the per-component comparison in Table 2 is what replaces them. We reuse the LoRA-Guard recipe not to beat these guards on a leaderboard — we score four small open checkpoints, not the gated production guards — but to isolate what the fine-tune itself does. GuardBench and its peers are the backdrop against which Section 3’s question is posed: they normalize the single-suite comparison whose fragility this report measures.

A.2Fine-tuning degradation and policy / benchmark transfer

The closest prior evidence is that fine-tuning can weaken rather than strengthen safety. Hsiung et al.^[25] show that safety guardrails can collapse after fine-tuning and attribute the collapse to similarity between the alignment and fine-tuning data — a mechanism that rhymes with our specialization finding, but is measured on a model’s own alignment behavior rather than on a guard classifier’s represented-versus-transfer ranking split. Liu et al.^[37] document that guardrails overfit their training policy and propose augmented policy training as a remedy; Li et al.^[34] give a complementary mechanistic account, showing that prompt-attack defenses learn surface heuristics that do not generalize — essentially a why for the transfer loss we observe empirically. Bassani and Sanchez^[5] probe the same fragility from the perturbation side, measuring guardrail robustness to input mutations and adversarial attacks, and Hackett et al.^[22] demonstrate concrete evasion of injection/jailbreak detectors. Framed most generally, Akinrele and Gowda^[2] argue that prompt-injection detection is regime-dependent — performance is a function of the deployment regime, not a fixed model property — which is our thesis stated for one task with interpretable structural signals.

Our addition is the measurement design rather than the qualitative claim. Where these works compare a model before and after, or one policy against another, we hold the checkpoint, manifest, seeds, and scorer fixed and read the paired change as a distribution over a purposively chosen panel with a hierarchical bootstrap, and we decompose it into a large represented-source gain versus a near-flat but heterogeneous transfer change (Table 3, Figure 4). Crucially, we do not stop at “degradation”: we quantify the specialization geometry per checkpoint and per benchmark, carry it to a deployable operating point (Table 5), and then ask whether a composition (Section 5) changes it. And unlike Liu et al.^[37], whose remedy retrains with augmented policies, our candidate remedy retrains nothing.

A.3Calibration and operating points

Ranking quality (average precision) and thresholded decision quality are distinct, and the gap between them is a calibration problem. Guo et al.^[21] established that modern neural networks are systematically miscalibrated and that simple post-hoc scaling helps; Liu et al.^[36] specialized this to LLM-based guard models, showing they are poorly calibrated for reliable content moderation; and FlexGuard ^[13] moves past a single fixed threshold toward continuous, strictness-adaptive risk scoring. This literature motivates two choices we make: we fit calibrators only on a development split before reading any threshold, and we report operating-point behavior (macro- and pooled-FPR, TPR, single-class recall) separately from ranking.

Our specific contribution to this thread is a clean separation of the two failure modes on the same guards. In Act II we show that a composition can recover transfer ranking while its realized false-alarm rate at a fixed FPR target still misses (Table 14): recovering rank does not deliver a transferable threshold. In Act III the same lesson recurs from the other direction — for tightly clustered guard scores the fixed-threshold operating point is knife-edge and unstable across software versions, which is why we deliberately do not tabulate it there. Where the calibration literature asks “is this score a probability?”, we use that machinery to make the sharper point that ranking recovery \(\neq\) calibration transfer, a distinction a benchmark AP number alone hides.

A.4Weight-space versus output-space composition

Combining models to improve robustness under distribution shift is well studied in weight space: WiSE-FT interpolates a fine-tuned model with its zero-shot initialization ^[53], and model soups average the weights of several fine-tunes ^[52], both trading a little in-distribution accuracy for out-of-distribution robustness at no extra inference cost. The theoretical companion most relevant to us is Kumar et al.^[31], who show that calibrated ensembles — combining models in output space after calibration — can mitigate the accuracy–robustness tradeoff under shift; this is the justification behind our composition rule.

Act II (Equation 5) is deliberately the output-space cousin of these methods: we average the base’s and the adapter’s calibrated scores with fixed equal weights, retraining nothing. The tradeoff versus WiSE-FT/soups is explicit — output-space composition needs only comparable scores, not interpolable weights, so it is portable across checkpoints that could never be weight-averaged, but it pays for two inference passes. We position composition as a transfer-recovery remedy for guard specialization specifically (represented cost small, transfer recovered relative to SFT; the aggregate edge over the base is nominally positive but sits inside the reproduction envelope of Section 3.7 and is unresolved — Table 11 and Table 12), and we are explicit about what the pilot does not include — an actual WiSE-FT weight-space rescoring is left as a control in the roadmap, and the logit-average variant is reported only as a non-promotable ablation. The equal-cost SFT+SFT control is run (Table 13): base+SFT beats SFT+SFT on every checkpoint, so the transfer recovery is attributable to keeping the base rather than to generic two-model ensembling. To our knowledge this calibrated output-space average has not been evaluated as a targeted fix for the represented/transfer split in small safety guards.

A.5Domain-specialized guarding and regulated verticals

A recent line moves guarding from generic harm to domain compliance. ExpGuard ^[9] provides expert-annotated content moderation in specialized domains including finance, health, and law — we adopt it directly as our external, expert-labeled breadth replication (Table 17; complete four-checkpoint base result). FinGuard ^[14] detects financial regulatory non-compliance in LLM interactions; MortarBench ^[51] evaluates mortgage loan-origination agents; and Bowen III et al.^[6] measure and mitigate racial bias in LLM mortgage underwriting. Adjacent security-benchmark methodology such as Gate AI ^[20] rounds out the evaluation-design context, and our mortgage instrument is grounded in public HMDA loan-level data ^[16].

Each of these captures one facet our four-domain design tries to unify while keeping the honesty stance explicit. FinGuard is single-domain and, like the general guards, reports absolute detection rather than a paired base-versus-tuned transfer delta. MortarBench evaluates whether an agent completes an origination task, not whether a guard detects a compliance violation, and it does not isolate the requests that read as safe yet violate policy. Bowen III et al.^[6] study bias in the model’s own underwriting decisions, whereas we study a guard’s detection behavior and add a protected-class minimal-pair invariance gate as a fairness signal that ranking alone misses. Our mortgage benchmark’s distinctive construct is the dual label \(G\times D\) (Table 15, Figure 17), whose load-bearing G0/D1 stratum — looks safe, is a violation — is exactly the case a general-safety score cannot see; the zero-shot baselines (Table 16) show these compliance violations are only moderately ranked even by the strongest base. Two honesty boundaries separate our instruments from the certified-audit reading a reader might infer: the mortgage labels are LLM-judge, policy-card-consistent — not SME-adjudicated, and ExpGuard supplies the strictly stronger expert-labeled tier but as a single-label transfer probe, not a dual-label construct. We therefore pair one domain built in depth (mortgage, dual-label plus fairness gate) with three external verticals for breadth (finance/health/law via ExpGuard), all scored on the same fixed panel used in Acts I–II so the specialization question can be asked identically across domains.

The component-by-component comparison against the six closest contributions, and the narrowed novelty statement it forces, are in Section 1.2 and Table 2 — placed in the Introduction beside the claim they bound.

BThe shared experimental setup

The paired studies in this report — SFT, composition, and the zero-shot domain baselines — reuse one fixed, purposively chosen panel of four checkpoints and one frozen, decontaminated 1,200-row training manifest, scored by the identical single-token \(z_{\text{unsafe}}-z_{\text{safe}}\) head (Equation 1) and the same tie-aware macro-AP, under fail-closed provenance locks that bind the data, code, and software versions and refuse to run if any fingerprint mismatches. This single-panel, single-manifest discipline is what makes SFT, composition, and the domain evaluations directly comparable rather than stapled results. The same four checkpoints recur across all three acts, and Qwen3-4B is the recurring character: the strongest base, and — as the acts will show — the one that specializes most, the one composition helps least, and the numerically highest-ranking zero-shot mortgage guard (within overlapping CIs — Table 16 does not license a ranking). It is not the most protected-token-invariant guard on that split: its \(\Delta_{\mathrm{context}}=0.000\) is a saturation artifact of the probability scale, and read on the raw margin its gap is the second largest on the panel (Section 6.3).

B.1The fixed four-checkpoint panel and pinned revisions

The panel spans two model lineages and \(1.5\)–\(4\)B parameters: Qwen2.5-1.5B-Instruct, SmolLM2-1.7B-Instruct, SmolLM3-3B ^[3], and Qwen3-4B. Each checkpoint is pinned to a fixed upstream revision, and for each we verify that safe and unsafe map to distinct single tokens under a fixed convention (leading-space " safe"/" unsafe" first, no-leading-space fallback), recording the token IDs and strings. Table 26 lists the panel and the frozen recipe together. The estimand is this exact four-checkpoint panel; results are not extrapolated to other scales, architectures, or vendors.

B.2The single-token guard formulation and prompt rendering

Every base and every adapter is scored by the same guard formulation from Section 2.1: the prompt is wrapped in one versioned instruction template asking for a one-word verdict, and the last position’s safe/unsafe logits are read and stored as \(s(x)\) (Equation 1), with the softmax probability (Equation 2) derived from them. The semantic system instruction is shared verbatim across all four checkpoints, but each model’s own chat template renders it differently, producing three distinct rendered-template fingerprints across the panel; each is recorded. Long user content is budgeted and truncated before final rendering, and the runner then asserts that the complete classification instruction and wrapper survive the truncation — a contract check motivated by an earlier artifact that could left-truncate the instruction itself. Truncation strategy and scored token counts are stored per row.

B.3The frozen LoRA-SFT recipe

The recipe (lower panel of Table 26) is fixed identically across all four bases so that the only thing varying within Act I is the checkpoint. It uses a completion-only loss — the cross-entropy is applied only to the verdict token and an appended end-of-sequence marker, not to the prompt — with a LoRA adapter of rank \(r{=}32\) and scaling \(\alpha{=}64\) (dropout \(0.05\)) inserted into the attention and MLP projections (q, k, v, o, gate, up, down). Training runs \(300\) optimizer steps at learning rate \(2\times10^{-4}\) on a cosine schedule with \(0.03\) warmup and an effective batch size of \(4\), at maximum sequence length 1,024. At effective batch \(4\), the 1,200-row manifest yields exactly \(300\) updates — one complete exposure of the data. Each base is adapted with five training seeds (\(42\)–\(46\)) that share one data-order seed (\(42\)), giving \(20\) adapters; the four untuned bases are scored once and reused, for \(24\) model bundles in all.

B.4The 1,200-row training manifest and exclusions

Training reads one frozen manifest and never resamples a live dataset mid-run. It draws \(400\) rows each from three represented sources — ToxicChat ^[35], Prompt-Injections, and Jailbreak-Classification — with \(200\) safe and \(200\) unsafe rows per source, selected by a hashed rank salted with the data seed and frozen as one shared row order across every checkpoint and seed (Table 27). Two datasets are deliberately excluded from training: BeaverTails ^[28], because its safety annotation labels the prompt–response interaction rather than the prompt alone, and OR-Bench ^[11], which is reserved for the benign stress set; including either would confound the prompt-only, dataset-held-out design. The study uses the non-commercial academic data branch (ToxicChat is CC BY-NC 4.0), so any released adapter inherits a non-commercial mark, and no third-party raw text is redistributed — only pinned identifiers, revisions, and content hashes.

B.5Three evaluation regimes and decontamination

Evaluation is partitioned into three regimes, forming a spectrum of how “new” each test is to the guard (Figure 15 shows the full construction, from sources through the family-isolation gate to these regimes; Table 28 is the complete roster of every dataset used in this report, with its role, size, and reference).

• Represented-source \(=\) {ToxicChat, Prompt-Injections, Jailbreak-Classification} rows held back from training. These measure discrimination on sources the guard trained on.

• Dataset-held-out transfer \(=\) {JailbreakBench ^[7], XSTest ^[50], WildGuardTest ^[23], WildJailbreak ^[29]}, whose rows were withheld from SFT. Because these encode heterogeneous native constructs (harm, refusal, jailbreak, contrast), their macro-average mixes distinct policies; we study dataset-held-out benchmark transfer, not one fixed universal policy.

• Stress \(=\) {OR-Bench-Hard benign portion (false-positive rate only), HarmBench ^[39] (recall only)}. These are single-class probes: no AP or AUROC is computed on stress data; they only watch for over-blocking (benign FPR) and missed attacks (recall).

The primary metric within each regime is benchmark-macro AP; the secondary metric is the calibration-targeted operating point of Section B.6.

Decontamination (family-isolated splits, MinHash).

To keep near-duplicate rows from straddling the train/evaluation and calibration/test boundaries, the builder first preserves the authoritative upstream conversation, pair, and scenario identifiers, then adds deterministic character \(5\)-gram MinHash edges between near-duplicate rows. It forms the connected components of that graph — “families” — and assigns whole families to a single split, so that no family can appear in both training and any reported evaluation, or in both calibration and any reported test or stress surface. The build fails closed if any family crosses a forbidden boundary. This family graph also supplies the Poisson(1) re-count weights used by the bootstrap (Section 2.6). An independent audit of \(24\) hard assertions covers the source exclusions, schema and row identity, exact and conflicting-label overlap, pinned revisions and content hashes, selection provenance, family disjointness, licenses, and near-duplicate dispositions. This is a build-time gate (family isolation plus the \(24\) assertions), and it is distinct from the formal retrospective overlap audit of the manifest against the current v2 transfer suite. That audit has since been run (experiments/audit_overlap_lineage.py, committed to artifacts/overlap_audit/) and it is clean: against the 1,200-row manifest the v2 transfer suite shows zero exact and zero normalized matches, zero family_id or upstream_family_id collisions, nothing at \(5\)-gram containment \(\geq0.80\), and nothing at character-shingle Jaccard \(\geq0.70\), so the transfer estimates need no downward revision (Section E.1). What neither the gate nor the audit covers is semantic overlap: no embedding-space check is implemented, so a paraphrase sharing no \(5\)-gram with any training row would pass both, and closing that needs an encoder pinned into the lock (Section E.3).

B.6Estimands and the statistical protocol

For regime \(R\), checkpoint \(b\), and seed \(r\), the per-regime metric is the equal-weight macro-AP over that regime’s benchmarks,

using the tie-aware, non-interpolated scikit-learn AP. The paired base-to-tuned change for one cell is

and the fixed-panel aggregate averages the four per-checkpoint deltas without resampling checkpoint identities,

The headline object is the joint vector \(\theta=(\bar\Delta_{\text{represented}}, \bar\Delta_{\text{transfer}})\), kept as a two-dimensional quantity and read off the specialization plane of Figure 4 — horizontal axis the represented change, vertical the transfer change, four quadrants (uniform gain, specialize, uniform loss, transfer-favored) — rather than collapsed into a single “specialization score.” The axes and quadrant interpretation are result-independent.

Uncertainty.

We attach \(95\%\) two-sided intervals with 10,000 paired hierarchical-bootstrap replicates (fixed RNG seed \(20260712\)): checkpoint identities are held fixed, SFT seed indices are resampled within each checkpoint, and one Poisson(1) weight is drawn per recorded family (Section B.5). Leave-one-checkpoint-out and leave-one-transfer-benchmark-out values are deterministic sensitivity checks, not resampling or population inference. The locked analysis mode is precision-focused: we report estimates, intervals, and sensitivities, with no intersection–union test, multiplicity correction, bootstrap \(p\)-value, or pass/fail gate.

Calibration-only operating point.

As a secondary deployment diagnostic, we fit one positive temperature per bundle on the calibration split, then choose the threshold that maximizes calibration recall subject to a one-sided \(95\%\) Clopper–Pearson upper bound of at most \(5\%\) on pooled calibration-negative FPR; if no cutoff qualifies, the cell reports NO_FEASIBLE_THRESHOLD as an outcome rather than relaxing the target. Temperature and threshold are fit only on calibration rows — never on transfer or stress data — and the audit hard-asserts that calibration shares no family with any reported test or stress surface. The Clopper–Pearson bound assumes independent Bernoulli negatives, so it is a pooled diagnostic, not a family-aware production guarantee. The resulting operating-point rates appear in Table 5 (Act I) and Table 14 (Act II).

Matched-FPR points are ROC summaries, not deployable thresholds.

Three tables in this report — Table 6, Table 18 and Table 22 — place every guard at a common \(5\%\) false-alarm budget so that recalls are comparable. That budget is located by taking a quantile of the evaluated negatives and then measuring recall on the same rows. This is a legitimate and standard way to read a pair of ROC curves at one point, and the paired bootstrap re-estimates that functional inside every replicate, so the intervals are internally consistent. It is not a threshold a production system could use, because placing it requires the labels the system is trying to predict. The distinction matters because it changes what the number means: “TPR at an empirical matched-FPR ROC point” is a statement about ranking quality at a comparable alarm rate, whereas a deployment claim would require the threshold to be fixed on a disjoint calibration set, frozen, and then reported with its realized FPR and TPR on untouched acceptance rows. Only the calibration-only operating point above does that, and only for Acts I–II. No matched-FPR figure anywhere in this report should be read as an achievable deployment operating point, and none of the deployment guidance in Table 23 rests on one.

B.7The composition protocol

Act II’s remedy needs no retraining. For a single input we run the base and one SFT adapter, convert each raw score to a probability with a calibrator fit only on a development split, and report the fixed equal-weight average defined later as Equation 5. The \(\tfrac12\) weights are fixed in advance (never tuned on test rows), so the only cost is the second inference pass. This is the output-space composition of Section 2.8; the logit-average and convex-weight variants were visible during development and are reported only as non-promotable ablations.

B.8Domain evaluation instruments: mortgage and ExpGuard

Act III evaluates the same four checkpoints on two complementary regulated-domain instruments.

Mortgage (built in depth, dual-label).

We constructed a fixed, HMDA-grounded benchmark whose unit is one incoming request carrying two separately assigned labels, general-safety \(G\) and mortgage-policy \(D\) (Section 2.9) — separately assigned, but not empirically independent in v1, where the empty G1/D0 cell nests \(G\) inside \(D\) (Section 6.2). Requests are grounded in de-identified, banded HMDA fact sheets — never verbatim records — through an agentic pipeline that plans, grounds, generates, adversarially mutates, and rubric-bound-judges each item, accepting it only if its assigned label matches the target, before a provenance-tracked, content_family-isolated split into the frozen release (Figure 17; that grouping key is narrower than pair-level isolation, and one protected pair does cross a split — Section 6.2). The benchmark has \(994\) rows over four \(G\times D\) quadrants — G0/D0 (\(450\)), G0/D1 (\(502\)), G1/D1 (\(42\)), and G1/D0 (\(0\), empty by construction) — with the load-bearing G0/D1 stratum (reads safe, is a violation) the largest. It spans seven content strata (fair-lending \(204\), fraud \(112\), UDAAP \(90\), disclosure \(66\), ATR/QM \(54\), privacy \(18\), and benign \(450\)) and is split into train \(604\) / dev \(149\) / public-test \(146\) / private-test \(95\). As a fairness-invariance gate it embeds \(39\) protected-class protected pairs (\(78\) rows) differing only in the protected attribute, of which \(21\) are true single-token swaps and the rest substitute a longer phrase (Section 6.3); the induced prediction gap is denoted \(\Delta_{\text{context}}\). The row composition is in Table 15 and the zero-shot base results (AP\(\cdot\)G, AP\(\cdot\)D, \(\Delta_{\text{context}}\)) in Table 16 (Act III).

Finance, health, law (external breadth: ExpGuard).

To test whether the specialization/transfer pattern recurs across regulated verticals with expert labels — a strictly stronger labeling tier than the mortgage LLM-judge — we also score the same four checkpoints on ExpGuard ^[9], an expert-annotated moderation set of 2,275 rows spanning finance, health, and law, using its prompt_label for input-prompt classification (matching our task). This is a single-label transfer probe, not the mortgage dual-\(G\times D\) construct — one domain built in depth plus three external verticals for breadth. We report aggregate and per-domain AP for the four base checkpoints (Table 17, Figure 11). The four-checkpoint result is complete and reproducible from committed text-free per-row scores; the tuned arm and a hosted-frontier reference point have since been run on the same rows and are central to Section 7 (Table 18). What is still future work is a dual-labeled finance/health construct with expert sign-off (Section E.3).

CBenchmark construction and mechanism detail

C.1The attractor: post-SFT scores are benchmark-fixed, so “stronger bases specialize more” is arithmetic

The cleanest way to see what Act I does and does not show is to notice a regularity that runs underneath Table 3: fine-tuning behaves like an attractor. No matter where a checkpoint starts, SFT pulls its ranking to nearly the same score on a given benchmark. Represented-source AP after SFT is \(0.982 \pm 0.005\) across the four checkpoints; transfer AP after SFT is \(0.807 \pm 0.024\) — both far tighter than the base spread they came from (\(0.178\) and \(0.073\) respectively; the transfer variance shrinks about 9-fold). Figure 16 (left) shows the collapse: four widely spread base scores, one narrow post-SFT band.

Why this reframes “stronger bases specialize more.”

Once the endpoint is (approximately) fixed, the change we plot is pure subtraction: \(\Delta = \text{endpoint} - \text{base}\). A stronger base then must show a smaller \(\Delta\) — not because strong models are intrinsically fragile, but because they start closer to the ceiling. This is testable. Regressing \(\Delta\) on the base score across the four checkpoints gives slope \(-0.99\) (represented, \(R^2=0.999\)) and \(-1.05\) (transfer, \(R^2=0.91\)) — the \(-1\) the attractor predicts before any fitting. One caveat on reading those numbers: the represented \(R^2=0.999\) is close to definitional. When the endpoint band is as tight as it is on represented sources (\(\pm0.005\)), \(\Delta=\text{endpoint}-\text{base}\) is arithmetically almost a slope-\(-1\) line in the base score, so a near-perfect fit is nearly forced and is not independent confirmation of anything. The transfer fit, where the endpoint band is several times wider, is the load-bearing test — and it still lands at slope \(-1.05\). Either way the eye-catching ordering (SmolLM2 \(+0.040\) down to Qwen3-4B \(-0.150\)) carries no behavioral content beyond the endpoints themselves.

Where the real content lives.

Two places, both benchmark-owned rather than model-owned. First, where the endpoints sit: \(A_T\approx0.81\) is largely a property of the (training manifest, benchmark) pair — to the precision of the small residuals below, the base model has largely dropped out of the equation, and the post-SFT score is largely determined by the manifest and the benchmark rather than by which checkpoint you started from. This is the mechanism behind the title’s claim: after SFT the score is largely a property of the (manifest, benchmark) pair, so a gain measured on that pair says correspondingly little about any other. Second, the small residuals around the endpoint (at most \(0.027\) on transfer): these measure how much of a base’s identity survives the fine-tune, and are arguably a more honest per-checkpoint quantity than \(\Delta\) itself. LoRA only adds a low-rank correction in the base’s own features, so full base-independence is approximate by construction — the residual is exactly that approximation, made visible.

C.2HMDA grounding: de-identified, banded fact sheets

Realism is what makes the benchmark hard: a guard should face requests that read like genuine mortgage-workflow traffic, not toy prompts. We ground each scenario in the public HMDA 2022 loan-level snapshot ^[16], pulled from the FFIEC/CFPB Data Browser (the U.S. agencies that collect and publish HMDA data). Each source record is reduced to a banded, de-identified fact sheet: loan purpose, occupancy, banded loan amount, banded income, banded LTV and DTI, action taken, denial reason, and state — and never an exact dollar amount, a census tract, or any identifier.

Three properties make this PII-safe by construction. (i) Banding: every exact figure is replaced by a range (an income band rather than a dollar amount, likewise for LTV and DTI). (ii) Marginal sampling: each field is drawn from its own marginal distribution over the bands — how often each band occurs on its own — rather than from the real joint combinations of fields in any one record, so no actual borrower’s row can be reassembled. (iii) A build-time assertion that no emitted fact sheet reproduces a single source record verbatim; the frozen release carries contains_real_pii=false with zero violations at freeze. Following Bowen III et al.^[6], the fair-lending and ability-to-repay cells are deliberately biased toward borderline, higher-risk files (high DTI, high LTV, a prior denial) — exactly where underwriter discretion, and therefore bias, has room to operate.

C.3The agentic construction pipeline

Rows are authored and labeled by a five-stage agentic loop, summarized in Figure 17. The loop is deterministic in structure (the coverage cells it must fill are enumerated up front) but stochastic in surface wording (the LLM stages run at temperature \(>0\)), which is why the release is a frozen artifact rather than a regenerable one.

1. Planner. Enumerates coverage cells as a product of quadrant \(\times\) trap type \(\times\) policy card \(\times\) role \(\times\) protected context, so every intended combination is targeted rather than left to chance.

2. HMDA-grounder. Draws a de-identified banded fact sheet (Section C.2) matching the planned cell.

3. Generator. Authors the request in a mortgage-workflow role voice — applicant, loan officer, underwriter, processor, broker, or adversary — so the phrasing matches who would plausibly send it.

4. Adversarial mutator. Composes label-preserving tactics that make a violation subtle and deniable without changing its ground truth: euphemism, coded proxy, business justification, and buried injection.

5. Rubric-bound judge. Assigns \(G\) and \(D\) by reading the request against the policy cards (Section C.4).

The loop closes on an accept-iff-target rule: a generated row is kept only when the judge’s assigned label matches the target the planner set for that cell; otherwise the row is retried or dropped. This is what ties the surface wording (adversarially mutated to be hard) back to a controlled label. Domain-independent harm (the jailbreak seeds behind the \(G{=}1\) rows) is drawn from curated seeds; the novel mortgage-policy content is LLM-authored. Accepted rows then pass provenance and decontamination, a content_family-isolated train/dev/test split plus a committed 95-row extra slice that is not sealed and has already been spent (Section 6), and are published as the frozen v1_hmda2022 release — the object the reproducible harness scores.

Each row is built around exactly one trap — the tactic that makes its label hard to read off the surface. direct states the ask plainly; business_justified wraps it in a business rationale; coded_proxy substitutes a stand-in (a neighborhood, a language) for a protected group; euphemism softens the wording; occupancy_temptation invites lying about owner-occupancy; buried_injection hides an instruction inside otherwise ordinary text; over_refusal_bait looks alarming but is in fact benign, so a good guard should not flag it; benign_info is a plain safe request; and minimal_pair is the protected-class counterfactual of Section 6.3.

C.4The 24 policy cards and the label rubric

The mortgage-policy label \(D\) is not a vibe — it is defined by 24 benchmark policy cards spanning six regulatory families: fair lending, ability-to-repay / qualified-mortgage (ATR/QM), disclosures, UDAAP, fraud, and privacy. Each card is a one-page rule with two machine-usable parts: an “intervene iff” predicate (the precise condition under which a guard should step in) and an authority pointer (the law or regulation the predicate rests on). The judge reads a request against the applicable cards and emits \(G\in\{\text{safe},\text{unsafe}\}\) and \(D\in\{\text{allow},\text{intervene}\}\); the composed \(\mathrm{final}\) of Equation 6 follows mechanically, together with an action lattice (the ordered menu of responses, from allow through warn to block) and a severity grade.

The honesty caveat is load-bearing and stated plainly: these labels are policy-card-consistent, assigned by an LLM judge — they are not SME-adjudicated. The judge is run rubric-bound against the applicable cards at temperature \(0\) with a three-sample majority vote; all 24 cards remain unsigned, and judge agreement is measured as self-consistency (re-running the same judge and checking it agrees with itself), not as a human inter-rater study. Two transparency gaps we flag rather than resolve here: the frozen artifact does not record the exact judge model and version, so its possible model-family overlap with an evaluated checkpoint cannot be ruled out as a source of correlated blind spots; and per-label self-consistency counts were not exported, so self-consistency is asserted, not tabulated.

DEnsembling: when does combining guards recover transfer?

Ensembling is a classic way to make a classifier generalize, so it is natural to ask whether it repairs the transfer loss that fine-tuning induces (Section 3). We test this retrospectively on the adaptation panel’s committed per-row margins (Section 4): 9 checkpoints across 5 families (excluding the degenerate Llama-Guard null cell), each scored under {unmodified base, SFT, KL-SFT} \(\times\) five seeds on the same rows. The one exception is the cross-model committee of Table 30, which is reported over the four general checkpoints and over all ten including the null cell, because a rank-averaged committee is defined by its membership and dropping a member would change the object rather than clean it. Every number is equal-family macro-AP on the raw logit margin, anchored to each checkpoint’s own unmodified base — identical estimand to the main text. Within a checkpoint the base and its adapters share one output head, so raw margins are scale-comparable in units; they are not comparable in dispersion, and averaging them raw is therefore not an equal-weight average of judgments. On this panel the base’s margins are \(2\)–\(4\times\) wider than its adapters’ (e.g. Qwen3-4B \(\mathrm{sd}=16.3\) vs. \(4.2\)), so a raw \(\tfrac12/\tfrac12\) average is effectively base-weighted — one reason the main text’s operator calibrates first (Equation 5). Across checkpoints margins are not comparable at all, so the cross-model committee (below) rank-normalizes first. This appendix is retrospective on the inspected panel, not a preregistered claim.

Ensembling a fine-tune with itself is only denoising.

Averaging the five SFT seeds into one guard raises transfer by a bootstrap-significant but small +0.025 (one-sided LCB +0.020) over a single SFT run — yet transfer still lands below the base (-0.053 vs. base; Table 29). Seed-ensembling KL-SFT behaves the same (-0.008). The reason is structural: every seed is trained by the same recipe on the same data, so all members share the same specialization bias. Averaging them cancels seed variance, not the shared bias toward the represented sources — so it polishes a specialized guard without un-specializing it.

Ensembling across the specialization axis recovers transfer.

The one ensemble that lifts transfer above the base is the base \(\oplus\) adapter combination — the same base-plus-adapter idea as Act II, but not the promoted operator of Equation 5: this row averages raw margins (the non-promotable ablation of Section 5.5) and its adapter member is a five-seed mean, so it costs six forward passes rather than two. On this panel it raises transfer by +0.014 over the base (one-sided LCB +0.008 \(>0\)), recovering +0.066 relative to the five-seed SFT ensemble (not the single-run SFT row printed in the same table), at a represented cost of -0.022 (95% CI \([-0.035, -0.013]\)). It is a recovery, not a Pareto win: for the strongest base the composed guard still gives back a little transfer (the Act II caveat, Section 5). Cost-matched, the effect is about half as large. Recomputing the honest 2-pass analogues on this same 9-checkpoint panel, as an equal-checkpoint mean (the tabulated row is an equal-family mean, so the two weightings do not coincide and the control below is read as a magnitude check, not a reproduction): base \(\oplus\) one adapter on raw margins gains \(+0.012\), and base \(\oplus\) one adapter calibrated — exactly Equation 5 — gains \(+0.008\), against the \(+0.014\) of the six-pass row as tabulated. So the qualitative conclusion (only crossing the specialization axis lifts transfer above base) survives at the promoted operator, but the magnitude does not: the cost-matched gain sits at the one-sided LCB the six-pass row reports, and no main-text claim rests on this appendix. A generated, bootstrapped row for the 2-pass calibrated operator is a stated gap (Section E.3). Figure 18 makes the distinction visual: the seed-ensembling arrows stay inside the “transfer below base” band, while the arrow to base \(\oplus\) SFT crosses into “transfer recovered.” Adding KL-SFT as a third member (base \(\oplus\) SFT \(\oplus\) KL-SFT) lands essentially on top of base \(\oplus\) SFT — no further gain. The active ingredient is diversity across the specialization axis: the un-tuned base makes off-distribution errors that are decorrelated from the tuned member’s, so averaging cancels them; two specialized members do not. This is measured directly, not merely inferred from the AP midpoint: on transfer rows the base’s per-row errors correlate only 0.422 with its own fine-tune’s, versus 0.851 between two fine-tune seeds — the base is far more complementary to the fine-tune than one fine-tune run is to another, which is exactly why base \(\oplus\) SFT recovers transfer while SFT \(\oplus\) SFT (below) does not.

A committee of fine-tuned guards does not generalize better.

The strongest form of the technique — a cross-model committee that rank-averages different guards on the same row — makes the point sharpest (Table 30). A committee of the SFT guards is worse than the single best guard on transfer (-0.014 over the four general checkpoints, -0.056 over all ten): SFT drives every member toward the same represented sources, so their held-out errors are correlated and averaging cannot cancel them — it only dilutes the best member. The committee of KL-SFT guards merely breaks even (+0.004 / -0.004), because KL regularization leaves the members less specialized and therefore more diverse — but that is the KL penalty doing the work, not the ensemble.

ELimitations, the evidence ledger, and the validation roadmap

This section is the report’s honest accounting. Everything above is a set of measurements; this section states precisely what those measurements do and do not license you to conclude, records the provenance and label tier of each body of evidence in one unified ledger, and lays out the concrete steps that would upgrade each estimate into something stronger. Two rules are enforced without exception and should be read as the spine of the whole report: (1) retrospective, estimation-only evidence is never pooled with external or prospective evidence, and (2) no number anywhere is promoted into a causal, universal, or fair-lending claim.

E.1What these results do NOT establish

Cross-cutting claims we do not make.

The following limits apply to every act, regardless of which axis is in view.

1. No causal claim. We observe that fine-tuning co-occurs with specialization (Act I, Section 3) and that composition co-occurs with transfer recovery (Act II, Section 5). We do not isolate a mechanism. In particular, the reading that “stronger bases specialize more” — suggested by the ordering from SmolLM2-1.7B (\(+0.0400\) transfer change) down to Qwen3-4B (\(-0.1499\)) — is a hypothesis drawn from four points, not a demonstrated law. It is also mechanically entangled with the metric: because the paired change is \(\Delta=\mathrm{AP}_{\text{SFT}}-\mathrm{AP}_{\text{base}}\) and every SFT guard converges to \(\approx0.98\) represented-source AP regardless of where its base started (\(0.4524{\to}0.9806\) for SmolLM2 up to \(0.8855{\to}0.9837\) for Qwen3-4B), \(\Delta\) is arithmetically coupled to the base level. A negative correlation between base AP and \(\Delta\) is partly a definitional artifact and cannot be read as evidence of a behavioral tendency.

2. No population or universal claim. The panel is a fixed, purposively chosen set of four checkpoints drawn from only two model lineages (Qwen: Qwen2.5-1.5B, Qwen3-4B; and the SmolLM family: SmolLM2-1.7B, SmolLM3-3B). Lineage, scale, and native alignment recipe are therefore confounded: any “size effect” we might read off four points is inseparable from “which of two families it came from.” Nothing here estimates a distribution over models, and the hierarchical-bootstrap intervals (e.g. the aggregate transfer change \(-0.0589\) [\(-0.0837,-0.0321\)]) are explicitly conditional on this panel, not population intervals.

3. Heterogeneous native policies are not controlled. Each base checkpoint arrives with its own native safety training and implicit policy. A paired base\(\to\)SFT delta therefore mixes “what our fine-tune did” with “what this vendor’s alignment already did,” and the four bases do not share a common definition of unsafe. This is a feature for the paired design (we always compare a model to itself) but a limit on interpretation: the cross-checkpoint spread partly reflects incompatible starting policies, not just differing tunability.

4. Not confirmatory — the benchmarks were inspected during development. The 1,200-row training manifest and the transfer suite were visible to the researcher while the method was being built. “Held out” throughout this report means dataset-held-out (rows/sources not used in training), not sealed-from-the-researcher. This makes Acts I and II retrospective and estimation-only. No pre-registration protects them; the honest status is “a reproducible characterization of this panel,” not “a confirmed finding.” The one partial exception is the starting-type adaptation study (Section 4), whose estimands, decision rules, and non-inferiority margin were fixed in a committed claim registry (artifacts/starting_type_adaptation_v1/protocol/claim_registry.json) before any score existed — so it is analysis-preregistered. It is not data-blind: it re-scores the same 3,308 rows from the same frozen manifest as Acts I–II, the registry is finalization_status: dev_nonfinal, and no release lock binds it.

5. Decontamination against the v2 transfer suite is now audited, and it is clean — but the audit is not complete. The manifest was decontaminated during construction and reported as clean-v2; the formal overlap audit has since been run (experiments/audit_overlap_lineage.py, committed to artifacts/overlap_audit/) and found no leakage from the 1,200-row training manifest into the v2 transfer suite: zero exact matches, zero after normalization, zero family_id or upstream_family_id collisions, zero rows at \(5\)-gram containment \(\geq 0.80\), and zero at character-shingle Jaccard \(\geq 0.70\) (worst per-source Jaccard \(0.139\) jailbreakbench, \(0.101\) wildjailbreak, \(0.214\) xstest, \(0.538\) wildguardtest). The transfer estimates therefore require no downward revision. Two gaps remain and are the reason this is not struck from the list. First, Section 7.4’s represented splits are not clean in the same sense — id_test is held out by row, not by content, and \(1.6\%\)–\(5.0\%\) of each represented split sits within Jaccard \(0.70\) of a training row (\(2/67\) prompt_injections, \(8/159\) jailbreak_classification, \(7/451\) toxicchat), which is expected for held-out rows of a represented source and templated jailbreak corpora but does qualify those margins. Second, the audit implements exact, normalized, \(n\)-gram, near-duplicate and provenance-lineage checks but not an embedding-space check, which needs a pinned encoder whose identity would have to enter the lock; a semantic near-duplicate that shares no \(5\)-gram would still pass.

6. Single recipe, single manifest. Act I uses exactly one LoRA configuration (rank 32, \(\alpha{=}64\), dropout 0.05 on q,k,v,o,gate,up,down; 300 steps; lr \(2\!\times\!10^{-4}\) cosine, warmup 0.03; effective batch 4; max_len 1024; seeds 42–46 with data order fixed at seed 42) on one frozen manifest. The specialization signature is a property of this recipe on this data; a different rank, step budget, or data mixture could move the represented/transfer trade-off, and we do not sweep them.

7. Balanced-prevalence evaluation overstates production precision. The transfer regime is scored on a balanced pool (790 unsafe vs. 790 safe rows) and the represented regime is near-balanced (313 vs. 364). Real inbound traffic is overwhelmingly benign, so unsafe prompts are rare. Average precision and any precision-at-threshold measured on a balanced set are optimistic relative to a deployment where the negative class dominates: at low base rates the same ranking yields far lower precision. This is now made quantitative rather than left as a caveat — Section 3.8 and Figure 5 recompute \(\mathrm{AP}(\pi_+)\) exactly from the ranking (Equation 4): the base transfer guards fall from \(0.79\)–\(0.94\) balanced to \(\approx0.11\)–\(0.56\) at \(1\%\) prevalence (Figure 5), and the guard ordering itself re-spaces as positives get rare. Read every AP here as a ranking quality on a balanced pool, not as a deployed precision.

8. Ranking recovery is not threshold transfer. All AP-based results are threshold-free; deployment is not. At a calibration-selected operating point the gap is stark: SFT lifts represented-source recall from \(13.0\)% to \(76.9\)% but raises the transfer false-alarm rate from \(8.1\)% to \(15.5\)% (macro) — and pooled FPR moves further, \(4.3\)%\(\to17.0\)% — while HarmBench recall falls from \(78.0\)% to \(60.0\)% (Table 5). Equalising the alarm budget removes the apparent recall gain entirely and reverses it on all four checkpoints (transfer recall \(0.517\!\to\!0.217\); Table 6), so the deployment cost of specialization is larger than the unequal-rate row suggests, not smaller. Composition recovers rank but its realized transfer FPR at a 5% target is \(11.4\)% — better than SFT’s \(15.5\)% yet still above target (Table 14). A better AP does not hand you a transferable cutoff. The size of these blow-ups is telling: a Gaussian-tail calculation maps the macro FPR shift (\(8.1\%\!\to\!15.5\%\)) to only \(\approx0.4\) standard deviations of calibration drift, and the pooled shift (\(4.3\%\!\to\!17.0\%\)) to \(\approx0.8\) SD — sub-sigma drifts that no realistic calibration protocol excludes, because a fixed cutoff’s false-alarm rate is exponentially sensitive to drift at the operating quantile.

9. Prompt-only scope. We classify the incoming request (binary safe/unsafe) via the single-token \(z_{\text{unsafe}}-z_{\text{safe}}\) head. We do not evaluate response classification, multi-turn context, tool-call arguments, or streamed content. Guards that read the assistant’s output or a full dialogue are a different task and are out of scope; nothing here should be read as a claim about them.

Act I (SFT specialization) — what it does not establish.

The headline aggregate transfer change of \(-0.0589\) is not a clean “fine-tuning barely hurts transfer.” It pools opposing per-checkpoint effects (Qwen2.5-1.5B \(-0.0389\) [\(-0.0829,+0.0062\)]; SmolLM2-1.7B \(+0.0400\) [\(+0.0003,+0.0776\)]; SmolLM3-3B \(-0.0869\) [\(-0.1114,-0.0613\)]; Qwen3-4B \(-0.1499\) [\(-0.1963,-0.1050\)]); the panel average is small precisely because it averages a gain against three losses. The specialization plane (Figure 4) shows \(15\) of \(20\) (checkpoint, seed) points in the specialize quadrant and \(5\) in uniform gain — a tendency on this panel, not a universal direction, and one seed-family (SmolLM2) runs the other way.

Act II (composition) — what it does not establish.

Composition (Equation 5) recovers transfer relative to SFT (\(+0.076\) [\(+0.058,+0.093\)]) at a small represented cost (\(-0.019\) [\(-0.031,-0.010\)]), landing nominally above the base on transfer (\(0.883\) vs. \(0.866\) — an edge inside the \(0.015\)–\(0.029\) reproduction envelope of Section 3.7, and therefore unresolved). But: (a) it is not Pareto dominance — versus base it is heterogeneous, helping the weaker bases (SmolLM2 \(+0.067\), Qwen2.5 \(+0.036\) vs. base transfer) while hurting the strongest, Qwen3-4B (\(-0.030\)); (b) the mechanism is unproven (the ensemble/diversity account is a motivation, not a proof); (c) one control is run and one is still missing — we have run the equal-inference-cost SFT+SFT baseline (Table 13: base+SFT beats SFT+SFT on all four checkpoints, attributing the recovery to “keeping the base” rather than to “ensembling anything”), but not a true weight-space WiSE-FT rescoring ^[53], so we cannot say which family recovers more transfer here; and (d) the logit-average and convex-weight variants (transfer \(0.891\) for logit-average) were visible during development and are reported only as non-promotable ablations. Whether composition also recovers transfer for guards tuned with other objectives is left open.

Act III, mortgage — a measuring stick, not a legal finding.

The mortgage benchmark’s 994 dual labels (\(G\)=general-safety, \(D\)=mortgage-policy) are assigned by an LLM judge against written policy cards, with self-consistency checks but no subject-matter-expert (SME) adjudication and no human inter-annotator agreement (no Fleiss-\(\kappa\)). It therefore surfaces guard behavior on the load-bearing G0/D1 stratum (reads safe, is a violation); it certifies nothing about any real lender, applicant, or model, and licenses no fair-lending or disparate-impact conclusion. Four further limits bound it: (i) the G1/D0 quadrant is empty (0 rows), so “looks unsafe yet is policy-compliant” — the over-refusal-on-compliant-requests failure mode — is unmeasurable here; (ii) several strata are small (G1/D1 = 42 rows; private_test = 95 rows; 39 protected pairs / 78 rows, of which only 21 are true single-token minimal pairs and 38 have both arms in one split), so per-stratum estimates are noisy; (iii) the zero-shot bases rank policy violations only moderately (AP\(\cdot\)D \(0.67\)–\(0.85\); Table 16) — and against a chance floor of \(0.555\) (81/146 \(D\)-positives) that band is only \(0.12\)–\(0.30\) above chance, so much of the subtle G0/D1 stratum stays unresolved by ranking. \(G\) and \(D\) are also not independent in v1: the empty G1/D0 cell makes \(G\) a strict subset of \(D\) (\(\phi=0.19\)), which is a property of the release, not of guard behaviour; (iv) the fixed-threshold operating point is deliberately not tabulated because it is knife-edge for these clustered-score guards (its G0/D1 catch count swung by more than 50 rows across library versions) — itself a finding about unreliable threshold transfer, not a number to deploy. The protected-pair gap \(\Delta_{\mathrm{context}}\) is a fairness signal, not a fairness verdict — and on this split it does not rank guards: Qwen3-4B’s \(0.000\) is a saturation artifact (\(0.797\) log-odds on the raw margin), and Qwen2.5-1.5B’s \(0.183\) is carried by one pair that contrasts a named trait against a seven-word placeholder rather than swapping a single token (excluding it, \(0.020\)). See Section 6.3.

Act III, ExpGuard — external, single-label, never pooled.

The finance/health/law replication uses external, expert-annotated labels (prompt_label) over 2,275 rows — a strictly stronger labeling tier than the mortgage LLM-judge. It tests whether the specialization/transfer pattern recurs across regulated verticals, but it is single-label (not the mortgage dual \(G\times D\) construct), so it is a breadth/transfer probe, not a compliance construct. The four-checkpoint base result is complete (Table 17) and reproducible from committed per-row scores. The tuned comparison and a frontier-model reference point are now also present (Table 18, Section 7), with one provenance caveat that governs every tuned row: the Act I release adapters no longer exist — they were produced on an ephemeral runner whose bucket was deleted at cleanup — so those rows are the KL-SFT \(\beta=0\) arm, a distinct execution of the same recipe under the same LOCK contract and train manifest, with different adapter_sha256 values. They are labelled SFT (in-env) wherever they appear, they are the same sft_inenv quantity klsft_summary.json already reports beside sft_committed, and no Act I headline number is restated from them. What remains genuinely open is a dual-labeled finance/health construct with expert sign-off. Because ExpGuard’s labels come from a different source and tier, its numbers are never averaged, ranked, or otherwise pooled with the mortgage LLM-judge numbers or with the retrospective panel.

E.2The unified evidence ledger

The report carries evidence at different tiers, and the honesty of the synthesis depends on never letting a weaker tier borrow credibility from a stronger one. Two dimensions matter. The first is the evidence flavor: is the number retrospective / estimation-only (measured after the fact on inspected data, conditional on a fixed panel) or does it come from an external, independently annotated source? The second is the label tier, in decreasing strength: SME-adjudicated with reported inter-annotator agreement (the gold standard — we have none of this yet); external expert-annotated (ExpGuard); and LLM-judge, policy-card-consistent (the mortgage benchmark — self-consistent against written cards, but no human). Table 31 records, for each body of evidence, its flavor, its label tier, what it establishes, and what would upgrade it. The governing rule is stated once and applied everywhere: the three columns of flavor are never pooled. Retrospective panel numbers (Acts I–II), the LLM-judge mortgage numbers (Act III depth), and the external expert-annotated ExpGuard numbers (Act III breadth) are reported side by side but never averaged into a single headline.

E.3The validation roadmap

Each limitation above has a matching upgrade. The roadmap is ordered roughly by how much it would strengthen the central claims, and every item names the specific weakness it closes.

1. Lock a genuinely uninspected prospective cohort (closes “not confirmatory,” Acts I–II). Pre-register the estimands and decision rules, then evaluate on data neither the researcher nor the manifest builder has seen. This is the single upgrade that would move Acts I and II from “retrospective characterization of this panel” to a confirmatory finding; the adaptation study (Section 4) applies the preregistration half of this discipline (locked estimands and gates) to the starting-type question, but not the uninspected-data half — it reuses the Act I manifest and rows — so the remaining step is a genuinely uninspected cohort for both it and Acts I–II’s core specialization estimands.

2. Add the embedding-space overlap check (the remaining half of the decontamination audit). The \(n\)-gram, near-duplicate, normalized, exact and provenance-lineage checks are now run and clean (Section E.1), so the pending-audit caveat is closed for lexical overlap and the transfer estimates stand. What is left is semantic overlap: a paraphrase sharing no \(5\)-gram with any training row would pass every check implemented. Closing it requires pinning an encoder in the lock — an identity decision, which is why it is a roadmap item rather than a script.

3. Add the remaining composition controls (closes Act II’s mechanism gap). The equal-inference-cost SFT+SFT baseline is now run (Table 13: base+SFT beats two-adapter ensembling on every checkpoint, so the recovery is the base’s doing); the open item is a true weight-space WiSE-FT rescoring ^[53] to compare output-space against weight-space interpolation on identical rows. A second, cheaper item belongs here: the ensembling appendix’s base \(\oplus\) adapter row is a six-pass construction, and its cost-matched two-pass analogue is quoted there as a hand recompute (Appendix D) — emitting that row through the generator with its own bootstrap would close the gap between the appendix’s tabulated magnitude and the promoted operator’s. A third, purely mechanical item belongs beside them: Figure 4 is the one figure in this report with no generator, so give make_figures.py a plotter that re-derives it from Table 4 and bring it inside the byte-check like every other figure (Section 9).

4. SME-adjudicate a stratified mortgage subset and report Fleiss-\(\kappa\) (closes the mortgage label-tier gap). Have qualified subject-matter experts independently re-label a stratified subset (weighted toward the G0/D1 and protected-pair rows), report inter-annotator agreement, and reconcile against the LLM-judge labels; this is what would let any mortgage number graduate from “measuring stick” toward an audit-grade instrument. Alongside it, score the fairness gate on all \(39\) release protected pairs rather than the \(3\) that fall inside public_test: the guards are zero-shot, so every pair is legitimate evaluation data and the recompute needs no training — and Section 6.3 shows three pairs cannot rank guards at all. Report it separately from the public-test pairs so the number stays comparable with a future fine-tuned arm.

5. Populate the empty G1/D0 quadrant (closes an unmeasurable failure mode). Construct grounded “looks unsafe yet policy-compliant” requests so over-refusal on legitimate mortgage traffic becomes measurable rather than absent by construction.

6. Evaluate at production prevalence with a cost-weighted operating point (closes “balanced prevalence overstates precision”). Re-score at realistic (heavily benign) base rates and report precision and a cost-weighted threshold, so the deployed-precision story is not read off a balanced pool.

7. Broaden the panel beyond two lineages (closes the lineage/scale confound). Add checkpoints from additional model families and a wider size range so that “stronger bases specialize more” can be tested rather than inferred from four coupled points.

8. Build dual-labeled finance/health constructs with expert sign-off (closes the remaining breadth gap). The four-checkpoint base ExpGuard eval is complete (Table 17), and the tuned and frontier comparisons are now in place (Table 18). What is still missing is the construct itself: extend the mortgage-style dual-label \(G\times D\) design into finance and health with expert adjudication — one domain built in depth plus verticals built for breadth. Two smaller follow-ons fall out of Section 7: re-run the tuned arm from release-provenance adapters if they are ever reconstructed (the present rows are a same-recipe re-execution), and score the KL-SFT \(\beta>0\) arms on ExpGuard, for which the adapter grid and the scoring path already exist.

9. Extend beyond prompt-only (closes the scope limit). The gated-guard half of this item is now done: the licenses were accepted, and six released purpose-built guards — including Llama Guard 3 and WildGuard — are scored on the Acts I–II rows through their native verdict contracts in Section 4 and on ExpGuard in Section 7.2, with the Llama Guard cell’s degeneracy diagnosed as our own harness bug (Section 7.2). What remains is the task boundary: everything here classifies the incoming request, and extending to response and multi-turn moderation is untouched. Note that several training sources are non-commercial / gated, which constrains redistribution — raw third-party rows are referenced by pinned identifier + revision + content hash, never redistributed.