Evidence is separated by what it can prove.

Experimental design

Three primary workflow families use real public records, live provider calls, distinct tool vocabularies, and exact source-grounded grading.

• 132 discovery and 30 held-out records per family, balanced across three exact classes
• Baseline, compiler, and macro order counterbalanced over all six permutations
• Quality computed independently of tool order from pinned source fields and excerpts
• Ten repeated test records used for structural and surface determinism analysis
• Paid provider outputs retained; later oracle corrections never rewrite online evidence
• Provider optimization overhead separated from held-out deployment measurements

Primary three-family result

Issue-type routing, PR-outcome audit, and backlog-attention routing cover 90 held-out public records. Compiled programs reach 90/90 exact outcomes versus 89/90 baseline; hand-written programs also reach 90/90.

The single backlog baseline miss is not a statistically meaningful quality improvement. Fair manual programs tie learned programs structurally on both new families. The evidence supports workflow-family transfer on one snapshot—not full-workflow cross-repository or time-forward generalization, and not runtime superiority over correct manual code. A separate, deliberately narrower two-read extension tests the cross-repository, time-forward question on its own terms (below).

Issue-type family detail

All 132 discovery traces choose the same three-read order, but the comment-limit argument varies and is not trace-groundable. GAC refuses the complete region and emits the longest valid prefix: record → labels. The ordinary agent performs the comments read and final rendering.

The compiler reduces requests 50.0%, total tokens 39.5%, observed wall latency 51.7%, and estimated cost 32.0% versus unchanged. The macro matches request reduction and is better on tools, tokens, and dollars. The paired latency interval between macro and compiler crosses zero; no runtime-superiority claim is licensed.

Cross-repository, time-forward extension

A separate frozen-source study asks the narrower question the primary families cannot: does a guarded artifact survive a different repository and a later time window? The task is simplified to an exact two-read PR-outcome contract, so this extends reach, not scope.

A provider-free preflight seals five repositories under a strict time-forward split. Executed discovery reaches 580/580 exact traces (116/116 per repository). Four repositories admit an artifact and complete 120 held-out paired records; pytorch/pytorch retires at compile time under the frozen-candidate exact gate. Baseline, compiled, and fixed-template conditions all pass 120/120.

On the frozen cohort the ungated fixed template is more efficient than the learned artifact, which sharpens rather than weakens the claim: the artifact earns its place through automatic discovery, guarded admission, and retirement—not by dominating hand-placed code on a simplified task. That cohort also keeps a principled negative. Across the four completed repositories the artifact compacts all 40 merged and all 40 closed_unmerged held-out pull requests and falls back on all 40 open ones.

A balanced rerun asks whether that open-only fallback is intrinsic. On the three repositories with enough class support for a balanced strict time-forward split (pandas-dev/pandas, psf/requests, pytorch/pytorch), a round-robin preflight seals 120 discovery and 60 held-out pull requests per repository, evenly split across open, merged, and closed_unmerged. All three admit the same two-read artifact, it compacts all 180 held-out records, and the verifier's pr.state hull contains both open and closed everywhere. The earlier fallback pattern and the pytorch/pytorch retirement are properties of the original frozen cohort design, not limits of the guarded runtime.

Prospective gate-frontier study: the pre-declared null

Every registered alpha=.05 gate this paper reports is a step function, so a pre-registered protocol committed a design — before any evidence existed for it — to test that at scale: the same five repositories above, 116 discovery and 60 held-out cases each, three arms compared under identical records, model, cache policy, and ordering. The third arm, a support-only comparator, is operationalized as alpha=1 in an otherwise byte-for-byte identical compile pass — same mining, synthesis, challenge, and frozen-candidate selection — so it isolates the risk budget specifically rather than testing a different mechanism.

Four repositories admit a candidate and complete 240 of the pre-registered 300 held-out pairs (580/580 exact discovery traces); pytorch/pytorch again retires at compile time, reproducing — on an independent, five-times-larger cohort — the same repository's retirement above rather than revealing a new failure mode. On those 240 pairs the learned gate and the support-only gate are statistically indistinguishable on every efficiency metric, because every admitting repository deploys the identical coverage-1.0 threshold.

Compilation depth is a safety variable

An earlier aggressive three-read artifact reduces provider requests by 75%, but records one compiler-only factual miss: 17/18 exact versus 18/18 for unchanged and macro. Clean tool replay therefore does not certify the model continuation. A provider-free continuation checker later detects and repairs that retained miss, but it is not a prospective live safety result.

Guarded composite and fair placement

GCS packages the admitted three-read program behind a continuation-pinned projection. Against a provider-visible macro on 12 fresh issues, both pass 12/12; GCS uses one rather than two provider requests, 38.9% fewer tokens, 40.0% lower observed wall latency, and 32.3% lower estimated cost.

A subsequent six-record study gives an independently authored manual program the same pre-model position. GCS and manual both pass 6/6 and tie at one request, one interface, and identical input tokens. Official GEPA 0.1.4 retains its seed after 14 task evaluations; its 59-request optimization overhead is reported separately. These are bounded, exploratory results—not general rankings.

Supplementary benchmark interoperability audit

Only NESTFUL and API-Bank retain complete observed values suitable for post-trace compilation. Eight other paths are preserved below for reproducibility, but they do not demonstrate optimizer value and are excluded from the main comparison.

Open the benchmark explorer to search and filter all thirteen rows — every substrate, execution status, effect breakdown, and claim boundary, generated from the evidence files.

Statistical interpretation

Efficiency comparisons are paired by issue. The artifacts report 10,000-sample paired bootstrap intervals and two-sided Wilcoxon signed-rank tests; binary quality uses exact McNemar tests. These secondary tests are descriptive and unadjusted. Zero observed failures is not equivalence: 0/30 still gives a one-sided 95% upper failure bound of approximately 9.5%.

Reproducibility

All quantitative tables and figures are regenerated from sealed JSON/CSV artifacts. Dataset revisions, source trees, checksums, task splits, native trace identifiers, model versions, and price assumptions are retained. The core verification path makes no provider calls.

.venv/bin/python paper/scripts/build_artifacts.py
.venv/bin/python paper/scripts/validate_artifacts.py
.venv/bin/python scripts/verify_release.py
.venv/bin/python -m pytest

Exact commands for the external benchmark paths are in the benchmark audit. The experiment verification report records the independent consistency checks and residual threats.