The Instrument That Could Not Be Built: A Pre‑Registered Capability Ladder Stopped at Its Own Calibration Gate

Dinand Tinholt

Independent research · 8 September 2026 · pre-registered in a ladder registration (6 September 2026) and a v2 registration (6 September 2026)

Executive brief: The Benchmark You Cannot Build · earlier study: the SIGIL campaign

1Why the ladder exists

The SIGIL campaign asked whether an AI supervisor authorizes differently when it carries a record of the damage its past approvals caused, and it found that consequence exposure pushes the supervisor into refusing more of the cases it should have allowed [1]. Every episode in that study ran on qwen3.5, a 35B-A3B mixture served without extended thinking, in both the worker seat and the supervisor seat. A reader who wants the result to go away has one good move. Maybe a weaker model panics where a stronger one would weigh the case properly, and the over-blocking is a property of that particular cognition rather than of the architecture wrapped around it.

The ladder registration was written to close that move off. It holds the prompts, the scoring, the cost constants, the generator and the report code byte-identical to the banked campaign state, and varies one thing: which model sits in both seats. Four rungs were registered, from qwen3:8b through qwen3.8:27b and deepseek-r1:70b to a 117B mixture, gpt-oss-120b. The registered grid drops the cells that carry no hypothesis weight and keeps tiers 4 and 5 under three oversight conditions at three seeds, 1,440 units per rung, 5,760 across the ladder. Power arithmetic for the primary contrast was worked out in advance and put the test at about 0.90 against the campaign's observed tier-5 gap.

None of those units exist. The ladder carries the same smoke gate that governed the original campaign, and the gate stopped it before the first campaign episode. What follows scores that stop from the raw ledgers, then scores the redesign that came after it, then works out what the two together say about whether an instrument of this shape can be built at all.

2Pre-registration statement

Two documents govern this report, both frozen before the probes they cover.

The ladder registration, 6 September 2026, registers the rungs, the grid, the power calculation, the fleet-safety rules, and a per-rung gate. A rung's campaign may launch only if its autonomous-only probe, 30 episodes per tier over five tiers on a tuning seed, shows overall error inside 10–35% and combined tier-4-plus-5 error at or above 5%, with errored units at or below 2%. On a failure the rung gets at most one retune, using only the registered difficulty dials, tuned on seeds at or above 100 and re-probed on seed 102. A second failure sends the rung to OUT-OF-BAND with its numbers published and its campaign cancelled. Three hypotheses were registered: H-L1 predicts that at every in-band rung the tier-5 blocked rate under the liability ledger exceeds the rate under plain review; H-L2 predicts that neither pre-commitment condition beats autonomous at tiers 4 or 5; H-L3 predicts that more than half of tier-5 blocks are wrongful.

The v2 registration, 6 September 2026, was written after the v1 ladder gated out and covers calibration only, with an explicit rule that no campaign launches under it. It adds qwen3.5 itself as rung R0, the anchor, on the ground that the instrument extending a published result has to keep the published model measurable. Its gate adds a gradient clause: error at or above 5% in at least three of the five tiers. The instrument counts as proven only if all five rungs pass all four clauses in the same round, with one generator state and no per-rung forks. Three rounds are allowed, each round one frozen generator state probed across the rungs, every round reported whatever it shows. If the anchor cannot be held in band in the round that holds the others, the design question returns to the principal rather than to a fourth round.

A non-merge rule sits across both: v2 results never pool with v1 results in any table or statistic. Run directories carry a v2- prefix and a metadata file in each records the generator variant, so pooling cannot happen by accident. This report keeps them apart.

3Methods

3.1 Probes, ledgers, gates

A probe runs the worker alone, no supervisor, across five task families that form the irreversibility gradient of the parent study: invoice approval, refund release, contract clause acceptance, data deletion, public correction. Ground truth comes from the generator, so nothing in the pipeline grades itself. Each probe writes 150 units to an append-only JSONL ledger with fsync per line, one 23-field record per episode, error-coded units included. Every probe used max_tokens 8192 and a 600-second timeout, uniform across rungs, so that thinking tokens could not manufacture truncations that differ by rung.

Both gates are mechanical scripts that read a ledger and print a verdict. I rescored all twenty-two probes from primitive fields with my own reader, which imports nothing from the harness, and compared against the verdicts logged at the time.

3.2 The two generators

Generator v1 is the frozen campaign state. A v1 episode carries four to six checks of which one or two sit near a boundary and the rest are slack, so whole-episode error tracks the slip rate on a single computation. Its one registered difficulty dial adds distractors.

Generator v2 was designed to fix what v1 could not reach, and its reasoning is worth stating because the results turn on it. If an episode's verdict depends on k exact micro-computations, each near its boundary, and a model gets each one right with probability a, then whole-episode accuracy runs at about a^k. The design argued that pushing k from roughly 1 to roughly 8–14 would compress a spread in a into a single band of episode error, and it built five mechanisms to get there: thresholds that depend on facts read by other checks, chained arithmetic, near-boundary margins on every check rather than one, facts that must be affirmatively recorded and are sometimes simply absent, and lists that have to be filtered or deduplicated before counting. Ambiguity was banned by construction, with an evaluator per family that reads exactly the fact dictionary the worker sees and recomputes the verdict, raising on any mismatch.

3.3 What moved between rounds

Round 1 ran the design's opening dial settings. Round 2 turned switch-like mechanisms down for the anchor's sake and turned volume and interference up hard for the strong rungs: longer invoice line sets, eight settled invoice numbers with two near-misses, three prior-refund decoys instead of one, five to seven open requests with near-miss identifiers, deeper origin-dedupe chains. Round 3 added two mechanism classes that rounds 1 and 2 left untested at full dose. Depth-2 interaction gives the exception to a rule its own precondition recorded in the case, so preferred-supplier status carries a validity date and a vendor name, premium membership carries a start date that must precede the purchase, and the liability floor moves from twice to three times annual value under a governing law outside an enumerated pair. Absence inside a present record gives a completed legal-hold check with no finding stated, and a consent filed under a near-miss spelling of the named party. Anchor-facing dials came down in the same state. Every change was logged with its rationale before the round's first probe, and the extended test suite ran green on the staged copy, including a balance check over 1,200 episodes and an evaluator-authority check over four seeds.

4Results

4.1 The v1 ladder: seven probes, seven failures, no campaign

Table 1 gives every v1 probe rescored from its ledger. The band wanted 10% and the probes returned between nothing at all and 2.7%. Two rungs produced literally zero errors in 150 units on the first probe, and qwen3.8:27b then produced zero again on its retune at the maximum registered dial, 300 units without a single mistake. The registered dial moved the ladder's best case from 0.7% to 2.7%, which is a quarter of the way to the floor of the band.

Table 1. Every v1 ladder calibration probe, rescored from the raw unit ledgers. All seven verdicts match the gate output logged at the time. The registered band is 10–35% overall with tier-4-plus-5 error at or above 5%. deepseek-r1:70b never received its allowed retune; the orchestrator finished after its first probe. An eighth probe, a 27b retune launched concurrently with the 70b probe, was aborted by the operator after two units on an API timeout and preserved separately as a contention artifact, then rerun cleanly as the calib2 row above.

4.2 The registered ladder hypotheses, scored

H-L1, H-L2 and H-L3 all read on in-band rungs, and the ladder produced none. The orchestrator log records the consequence at each rung in the same words: gate FAIL, out-of-band, campaign skipped. No campaign run directory was ever created for any of the four rungs. The ledger contract registered four campaign files at 1,440 units each and zero of those units exist.

Table 2. Scoring of the ladder registration against what ran. The ladder was stopped by the rule it registered for exactly this case, in the direction the rule calls saturation.

4.3 Generator v2: fifteen probes, three rounds, one verdict

Table 3 gives the full v2 gate table from the raw units. Round 1 put qwen3:8b in band and left both mid-size rungs at zero. Round 2 was the best state the campaign ever reached, holding qwen3:8b at 14.7% and deepseek-r1:70b at 15.3% with all four clauses passing on both, while the anchor sat one net error above the 35% cap and the two saturated rungs contributed one error between them across 300 units. Round 3 added the depth-2 traps, and the saturated rungs went back to zero.

Table 3. All fifteen generator-v2 probes, per-tier error in percent, rescored from the raw ledgers. Clause (a) is overall error inside 10–35%, (b) is tier-4-plus-5 at or above 5%, (c) is at least three tiers at or above 5%, (d) is errored units at or below 2%. Four probes pass all four clauses and no round passes all five rungs, which is what the instrument verdict requires. Every verdict matches the gate output logged at the time.

Figure 1 · Autonomous error by tier, five configurations, three rounds

Four probes pass all four clauses: qwen3:8b in all three rounds, and deepseek-r1:70b in round 2. The instrument verdict needs five in one round, so it is NOT PROVEN, and the anchor rule bites on top of that. qwen3.5 came in at 38.7%, 36.0% and 37.3%, above the cap every time, which under the v2 registration returns the design question to the principal rather than to a fourth round. Round 3 finished at 04:11 on 8 September and the log closes with the registered stop.

4.4 The failure signature: one shape, two directions

The live hypothesis going in was that a generator whose approvable episodes are near-boundary conjunctions would fail the same way at every rung, regardless of model capability. Half of that is right and the half that is wrong matters more.

What is flat is the kind of error. In round 3, refusals of approvable cases make up 88% of the anchor's errors, 80% of qwen3:8b's and 81% of deepseek-r1:70b's. Splitting by ground truth shows the same thing more sharply: the anchor errs on 63.6% of approvable episodes against 9.6% of the rest, qwen3:8b on 15.6% against 4.2%, deepseek-r1:70b on 16.9% against 4.1%. Every rung that errs at all errs three to seven times more often when the correct answer is yes. That is the conjunction working exactly as the mechanism predicts, since a reject verdict needs one violation spotted while an approve verdict needs every one of eight to fourteen boundary checks to come out right.

Figure 2 · Failure signature against model scale

What is not flat is the rate, and the spread is enormous. Round 3 runs 0.0%, 0.0%, 10.1%, 10.7% and 37.3% across the five configurations. Pooled over all three rounds, gpt-oss-120b made zero errors in 450 scored units and qwen3.8:27b made one, against 168 for the anchor over the same 450. That single 27b error was a seven-line invoice sum of $14,989 against a computed preferred-supplier tolerance cap of $14,990, a one-dollar margin. The conjunction does not level the rungs. It takes a difference in per-check accuracy and multiplies it, driving one configuration to near-total refusal and leaving two others untouched.

Model size does not order any of this. Spearman correlation between overall round-3 error and total parameter count is −0.03 across the five configurations, and −0.18 against active parameters. The one configuration above the band is also the only one served without extended thinking, and with a single case in that group I cannot separate the serving mode from the model identity. That reading is CANNOT-TELL from these data.

Figure 3 shows the collapse in a form that needs no scoring rule. At tier 2 the round-3 anchor approved nothing at all while the generator had made 60% of those cases approvable, and at tier 3 it approved nothing against a 50% approvable share. Thirty-three approvable episodes across the two tiers, thirty-three refusals. On the same episodes qwen3.8:27b and gpt-oss-120b reproduced the generated approve share to the point at every tier: 57, 60, 50, 43 and 47 percent against a ground truth of 57, 60, 50, 43 and 47.

Figure 3 · Approve rate by tier against the generated approvable share, round 3

One tier carries most of what keeps the two middle rungs in band. Drop tier 2 from round 3 and qwen3:8b falls to 5/120 and deepseek-r1:70b to 5/120, both 4.2% and both well under the floor, while the anchor stays at 31.7%. The refund tier's date-filter trap is doing most of the work, which is also why clause (c), the gradient clause, is the one that failed deepseek-r1:70b in rounds 1 and 3.

4.5 The arithmetic of one band across a capability range

The design's load-bearing claim was that composition compresses a capability spread. Work the algebra the other way and it does the opposite. If episode error is 1 − a^k, then landing inside 10–35% requires a between 0.65^1/k and 0.90^1/k, and the width of that window is about 0.325/k for k of any size. Depth 1 admits per-check accuracy anywhere from 0.65 to 0.90, a window a quarter wide. Depth 10 admits 0.9578 to 0.9895, a window of three points. Depth 27 admits 1.2 points. Raising k buys episode-level error out of a saturated model at the price of shrinking the accuracy window everything else has to fit into.

The fleet's accuracies come straight out of the approvable side. Averaging the design's declared depths over the round-3 approvable episodes gives a mean k of 10.3, and inverting a^k against measured approvable accuracy gives per-check accuracy of 0.9063 for the anchor, 0.9837 for qwen3:8b, 0.9822 for deepseek-r1:70b, and a lower bound of 0.9961 for the two rungs that made no errors at all. Table 4 turns those into the depth each rung would need, and Figure 4 draws the same thing.

Table 4. Composition depth each rung would need to sit inside the registered band, from its measured per-check accuracy. Zero-error rungs use the rule-of-three 95% bound on error, so their accuracy is a lower bound and their required depths are lower bounds too. The intersection across all five rungs is empty. So is the intersection of any two of the three groups: the anchor's window closes at 4.4 and the next window opens at 5.9, and the middle pair's window closes at 24.0 while the saturated pair's opens at 27.0.

Figure 4 · The gate, drawn in the space the generator can move

Depth cancels out of the feasibility question entirely. Since ln a = ln(episode accuracy) / k, and one generator state means one k for all rungs in a round, the ratio of log-inaccuracies across two rungs can be read straight off their approvable accuracies without knowing k. Holding both inside the band requires that ratio to stay at or below ln 0.65 / ln 0.90 = 4.09. Measured against gpt-oss-120b, the anchor's ratio is at least 21.5 in round 1, at least 17.6 in round 2 and at least 25.5 in round 3, using the rule-of-three bound each time, which makes each figure a floor rather than an estimate. The measured fleet is out by a factor of four to six on a quantity that no dial in the registered set touches.

5Verification

I rescored all twenty-two probes from primitive fields with a reader that imports nothing from the harness, recomputing per-tier counts, overall error, tier-4-plus-5 error, the gradient count and the errored-unit share, then compared each against the gate output written at the time. Twenty-two verdicts, twenty-two matches, including the four-clause breakdown on every v2 probe. Condition isolation holds: all 2,250 v2 units and all 1,050 v1 probe units carry condition: autonomous, with no supervisor field populated anywhere, so nothing in these ledgers touches the oversight contrast.

Seed hygiene holds. The v1 probes used tuning seeds 101 and 102, and the v2 rounds used 201, 202 and 203, one per round as registered. Campaign seeds 1 to 3 appear nowhere in any probe.

One check needs stating because it looks like a violation and is not. The v2 registration requires the generator's approvable share to sit inside 40–72% at every tier, verified over 300 episodes per tier before any probe. Read off the probe samples instead, at 30 episodes per tier, two of the fifteen tier cells fall below 40%: tier 4 at 36.7% in round 1 and 33.3% in round 2. Under a true rate of 52% the chance of a single 30-episode cell landing at or below eleven approvable is 6.7%, and across fifteen cells the chance of at least one is 0.65. The design-level balance check logged for round 3 reads 47.7–55.1% over 1,200 episodes. The probe-sample dips are sampling noise on the precondition rather than a breach of it.

6What this does to the published SIGIL result

Two readings survive contact with the data, and both belong in the record.

The refutation is untouched, and its worker-side mechanism gains company. No rung ever ran a review-versus-ledger contrast, so nothing here competes with the 30,000 episodes the published result rests on. More than that, the saturation finding lands on a condition the campaign had already measured. Tier 3 of the published campaign produced zero worker errors across 6,000 scored episodes, and the paper reports that its value measurements there are pure oversight cost, since a blocking supervisor facing a worker who makes no mistakes has nothing to buy and still pays the consultation fee. Four of the five ladder configurations reproduce that condition at every tier, not just at tier 3. If a stronger worker were dropped into the same architecture, review's value would start from a worse place, and H5's predicted crossover would have less room than the published campaign gave it. That is a reading rather than a measurement, and I mark it as one.

The generality question stays open, and the instrument for closing it does not exist. The capability-artifact objection is the reason the ladder was commissioned, and it is exactly as unanswered now as it was before. The ladder adds one fact that sharpens it: on this task family the study model is an outlier at the low end of the fleet. Pooled over the three v2 rounds it errs on 58.2% of approvable episodes where gpt-oss-120b errs on 0.0% and qwen3.8:27b on 0.4%, over 225 approvable episodes each. Whether a supervisor whose per-check accuracy sits near 1 would over-block under consequence exposure is CANNOT-TELL from anything measured here. The published paper's own limitation section already calls the result capability-local, and that wording holds, now with a measured reason and a named obstacle. The obstacle is section 4.5: the instrument that would settle it cannot be built by turning the registered dials, because the fleet's log-inaccuracy ratio exceeds the feasibility limit by a factor of four to six.

I have folded both readings into the unpublished SIGIL drafts rather than raising them as a separate correction, since neither draft has shipped.

7What this settles about a v3 generator

The standing question was whether to build a generator whose approvable episodes stop being near-boundary conjunctions. The ladder answers the first half cleanly and reframes the second.

Breaking the conjunction is necessary for the anchor. Its failure is an approvable-side failure and almost nothing else: 131 of 168 pooled errors are refusals, and 78% of its error mass lands on episodes the generator made approvable. Fix the conjunction and the anchor's error falls, which is the direction the band needs.

Breaking the conjunction does nothing for the rungs that stopped the ladder. qwen3.8:27b and gpt-oss-120b were correct on all 225 approvable episodes and all 225 non-approvable episodes each, across three generator states that added interaction depth, absence inside present records, near-miss identifier matching, origin deduplication and one-dollar arithmetic margins. Easier approvable episodes push them further under the floor. Their problem is that exact checklist compliance of this family sits at per-check accuracy near 1 for them, and the design bans the one lever that would change that, since the only way to make one check hard enough is to make it ambiguous, which is the confound that killed the first pilot.

So the question does not resolve to yes or no on a v3 generator. What the evidence closes off is the five-rung single-band instrument, at any composition depth, on this content family. What it leaves open has three shapes, and the first is already built. Round 2's frozen state held qwen3:8b at 14.7% and deepseek-r1:70b at 15.3%, both passing all four clauses in the same round, and that state is preserved. A two-rung ladder on it would test whether the over-blocking mechanism reproduces across two models that are not the study model, which is a weaker generality claim than the registered one and a real one. The cost is the thing to weigh: from the round-3 probe latencies, 1,440 units at the ladder grid works out at roughly 70 hours of wall clock for the 70B rung and 18 for the 8B rung, against a registered ladder budget of about 36 hours for four rungs. The second shape is a task family that stops being a checklist, which is a new design rather than a dial change and needs its own registration. The third is per-rung generator forks, which the ladder registration prohibits by name because it makes rungs incomparable, and I record it only to note that it was considered and is closed.

8Limitations

Thirty episodes per tier. A probe cell is 30 episodes, so a rung reading 0/30 at a tier has a 95% upper bound near 10% at that tier. The zero-error claims rest on pooling: 450 units per rung across three rounds, and 300 more for two of them under v1. At 0/450 the upper bound on gpt-oss-120b's overall error is 0.67%.

The per-check accuracy estimates depend on a declared k. Table 4 uses the design's stated composition depths, 7 to 13 by tier, averaged to 10.3 over the round-3 approvable episodes. If the true effective depth differs, every a in that table moves. The feasibility ratio in section 4.5 does not, because k cancels out of it, which is why I lean on the ratio for the conclusion and treat Table 4 as illustration.

One non-thinking configuration. The anchor is the only rung served without extended thinking and the only rung above the band. Those two facts cannot be separated with one case.

Round-to-round comparisons cross generator states. Each round is one frozen state and the states differ by design. Comparisons within a round are clean; comparisons across rounds carry the dial changes with them, which is why the feasibility ratio is computed within each round separately.

deepseek-r1:70b never got its v1 retune. The registered path allows one retune per rung and the orchestrator finished after that rung's first probe. Allowing a retune is not requiring one, so the discipline held, and the rung's neighbours at the same maximum dial came in at 0.0% and 2.7%, which brackets it. Whether a 70B retune would have crossed 10% is CANNOT-TELL, though a jump from 2.7% to the band would be four times anything the dial produced anywhere else.

No campaign ran, so nothing here bears on the oversight contrast. Every unit in this report is autonomous. The report characterizes an instrument and measures five workers, and it says nothing about supervisors.

9Reproducibility

Twenty-two probe ledgers, append-only JSONL, 150 units each, error-coded units included, one run directory per probe for v1 and for v2, with a metadata file beside each recording the generator variant and dial constants. Frozen artifacts: the ladder registration, the v2 registration, the v2 design document, the output contract (frozen 2026-08-30), the two gate scripts, the round log with every dial change recorded before the round it governs, the orchestrator log, and the round-2 generator state. Every number in this report comes from the unit ledgers and can be recomputed from the primitive fields with the four gate clauses in section 2.

10References

The registrations committed every round to ship whatever it showed. This report covers the outcome that occurred: the ladder stopped at its gate, the redesign stopped at its round limit, and the instrument is not proven.