# How many labelled cases a learned decision head needs

Source: https://dinand.com/research/how-many-labelled-cases-a-learned-decision-head-needs/
Author: Dinand Tinholt
Published: 2026-10-31

> A pre-registered dose-response curve for two learned decision heads on one synthetic onboarding process: the fine-tuned encoder reaches the walk’s ceiling at 1,000 labelled cases, the from-scratch classifier never does, and the shipped eight-vote deliberate mode is withdrawn.

A fine-tuned open encoder reached the walk's ceiling at 1,000 labelled cases, scoring 0.825 against a ceiling of 0.822 on the challenge set. A small classifier trained from scratch never got there, and at 3,000 cases it still sat 0.262 short.

Read the shorter version, with five charts, on [Medium ↗](https://medium.com/@tinholt/how-many-labelled-cases-a-learned-decision-head-needs-0be863325f24) or the executive post on [LinkedIn ↗](https://lnkd.in/p/gg8EDt_Z)

## The question

How many labelled cases does a learned decision head need before it matches what its workflow can deliver? Earlier work in this lab gave two points. An encoder fine-tuned on 363 onboarding cases beat the rule table and the judging head; the same recipe on 58 accounts-payable cases fell below the rule table. J9 registers the curve between them, and a second question with it: does the head's shipped deliberate mode, eight sampled votes per judgement question, earn its cost?

## What was registered

The design was frozen on 3 October 2026 after two adversarial reviews, one of the plan and one of the code. Every reading went on file as a number before any case ran: "reaches the ceiling" means the 95 % interval of the gap sits entirely below 0.05, and "level" means a point within ±0.03. The run launched on 11 October and closed at 07:09 on 14 October.

## What was measured

A new training split of 3,000 synthetic onboarding cases was cut into disjoint draws of 100, 300, 1,000 and 3,000 cases, with second draws at 100 and 300 to measure luck. On each draw two heads were trained, except the 3,000-case encoder, which failed: L, a from-scratch classifier, and LY, a fine-tuned open encoder. Each worked 200 challenge cases and a 200-case subsample of a fresh evaluation split, live and end to end.

The reference is the walk ceiling. The head sits inside an elimination walk, and with a perfect judge that walk gets 0.822 of decisions right on challenge and 0.795 on the subsample. A four-feature rule on surface cues, used as the walk's judge, scores 0.793 and 0.754, and an arm that lands within 0.03 of it is flagged CUE-ONLY. The rule table scores 0.565 and 0.553.

The run held 8,200 episodes with zero transport failures. One cell was cut: the 3,000-case encoder fine-tune hung in the GPU graph compiler at step 6,800, already projected at 337 minutes against a five-hour kill line.

## What was found

The encoder scored 0.642 at 100 cases, 0.713 at 300 and 0.825 at 1,000 on challenge; on the subsample, 0.634, 0.712 and 0.798. At 1,000 its gap to the ceiling on its own states was +0.005 [+0.000, +0.017] and +0.000 [+0.000, +0.000], so it reaches the ceiling on both splits. At 300 the gaps were +0.095 [+0.043, +0.155] and +0.086 [+0.038, +0.144], which falsifies the registered claim that 300 cases suffice.

At 100 cases the encoder already beat the rules on point estimate, +0.077 and +0.082, which falsified a second prediction.

From scratch, L ran 0.397, 0.501, 0.463 and 0.526 on challenge and 0.408, 0.544, 0.508 and 0.523 on the subsample. Its gap to the ceiling stayed between +0.183 and +0.396 at every size. From 1,000 to 3,000 cases it moved +0.062 [+0.000, +0.125] on challenge, crossing the +0.05 falsifier for a flat top, a step the same size as the 100-case draw spread of +0.061.

Eight sampled votes per question cost 44.08 model calls and 184 seconds per case on challenge, against 20.3 seconds for the single-call head. Its difference from a fresh single-call run was −0.010 [−0.039, +0.022] on challenge and +0.008 [−0.027, +0.045] on accounts payable, both inside the difference between two single-call runs. Under the registered rule the product claim is WITHDRAWN.

A follow-up registration put an untrained open decision model of the small tier in the head's seat. It scored 0.651 [0.603, 0.699] on challenge and 0.601 [0.550, 0.650] on the subsample: +0.086 and +0.048 above the rules, 0.175 and 0.197 below the encoder at 1,000 cases. On accounts payable it tied the judging head, −0.012 [−0.057, +0.033]. It also over-claimed less than the judging head: a wrong-claim rate of 0.235 against 0.355 on challenge, and 0.005 on the subsample.

## What did not work

Second draws moved accuracy by −0.013 to +0.123. At 300 cases on challenge, the encoder's second draw scored 0.807, a spread of +0.094 that matches the +0.095 gap the first draw left to the ceiling. So where the crossing falls between 300 and 1,000 cases rests on one draw per size; counted as contributing cases it sits between 180 and 606.

On challenge the encoder at 1,000 cases sat +0.016 [−0.005, +0.035] above the surface-cue rule on the same states, inside the registered margin, and the analysis flags it CUE-ONLY. On the subsample it sat +0.035 [+0.022, +0.049] above, outside it. The challenge set can't separate the encoder's judgement from that cue.

The deliberate mode's predicted gain of +0.04 lies outside its challenge interval and inside its accounts-payable one, which runs to +0.045. The encoder's top dose is missing, so whether more labels hold it at the ceiling is unmeasured.

## What it does not claim

Every number belongs to one synthetic onboarding process, scored against a programmatic oracle whose "done" and "blocked" classes model judges dispute, and one open-weight model family answers every model call in the harness.

## How it was verified

An independent agent recomputed every cell accuracy and every difference from the raw run directories with its own code, and opened the analysis output only after writing its own numbers. All 22 cell accuracies matched in scored and correct counts. Of 55 difference rows, 36 matched at three decimals; the other 19 differed by at most 0.007 because of which cases each side counted, and matched on the shared cases. The lexical-probe rows were not recomputed. The untrained model's numbers were recomputed the same way and matched the scorer to full precision, with one definitional difference in a wrong-claim rate.

## Open questions

Does the encoder hold the ceiling above 1,000 cases? Where between 180 and 606 contributing cases does it cross, once there is more than one draw per size? Would a head with real judgement show up on a test set that the surface-cue rule can't solve? And does the curve keep its shape on a process with dependent items, such as insurance claims?

## Reproducibility

The registration and its addenda predate every case. Code and data are available on request until the public repository opens.

Read the shorter version, with five charts, on [Medium ↗](https://medium.com/@tinholt/how-many-labelled-cases-a-learned-decision-head-needs-0be863325f24) or the executive post on [LinkedIn ↗](https://lnkd.in/p/gg8EDt_Z)

The full paper

Workflow-completion campaign · registration J9 · with the 16 October addendum

## How many labelled cases a learned decision head needs

A registered dose-response curve for two trained heads on one synthetic process: the fine-tuned encoder reaches the walk's ceiling at 1,000 labelled cases, the from-scratch head never does, and the shipped deliberate mode is withdrawn.

Dinand Tinholt · 30 October 2026
Design frozen 3 October 2026 after two adversarial reviews · launched 11 October · run closed 14 October 07:09 CDT with one cut cell · recomputed independently from the raw run directories: cell accuracies identical, differences within 0.007

Abstract

How much labelled history does a learned decision service need? The decision-head paper gave two points: an encoder fine-tuned on 1,567 decisions from 363 onboarding cases beat the judging head, and the same recipe on 201 decisions from 58 accounts-payable cases fell below a rule table. J9 registers a curve. On the onboarding process we trained two kinds of learned head, a from-scratch classifier (L) and a fine-tuned open encoder (LY), on disjoint draws of 100, 300, 1,000 and 3,000 labelled cases, and scored each live on 200 challenge cases and a 200-case subsample of the fresh evaluation split.

The reference is the accuracy the head's elimination walk reaches with a perfect judge, 0.822 on challenge and 0.795 on the subsample. The encoder reaches that ceiling at 1,000 cases on both splits and misses it at 300. It scores 0.825 and 0.798 at 1,000, with own-state gaps of +0.005 [+0.000, +0.017] and +0.000 [+0.000, +0.000], against 0.713 and 0.712 at 300, gaps +0.095 [+0.043, +0.155] and +0.086 [+0.038, +0.144]. The registered claim that 300 cases suffice is falsified on both. The from-scratch classifier never reaches the ceiling: at 3,000 cases it scores 0.526 and 0.523, gaps +0.262 [+0.188, +0.336] and +0.203 [+0.134, +0.278]. The 3,000-case encoder fine-tune hung in the GPU graph compiler and was cut. Second draws at 100 and 300 cases moved accuracy by −0.013 to +0.123; that spread is the curve's noise floor.

The shipped deliberate mode, eight sampled votes per judgement question, cost 44.08 model calls and 184 seconds per case. Against a fresh single-call control its difference was −0.010 [−0.039, +0.022] on onboarding challenge and +0.008 [−0.027, +0.045] on accounts-payable challenge, and its product claim is WITHDRAWN. A later registered run put an untrained open decision model of the small tier in the head's seat: 0.651 [0.603, 0.699] on challenge, above the rules and 0.175 below the encoder at 1,000 cases. Every cell accuracy matched exactly; 36 of 55 differences matched at three decimals and the other 19 differed by at most 0.007 through a case-set convention; the lexical-probe rows were not recomputed.

1,000

labelled cases at which the encoder first reaches the walk ceiling, on both splits (0.825 against 0.822; 0.798 against 0.795)

+0.262

the from-scratch classifier's own-state gap to the ceiling at 3,000 cases on challenge, interval [+0.188, +0.336]

−0.010

eight votes minus one call on challenge, [−0.039, +0.022], at 44.08 calls and 184 s per case; claim WITHDRAWN

The design was frozen on 3 October after two adversarial reviews, one of the plan and one of the code.

## 1Question

How many labelled cases does a learned head need before it matches what the walk can deliver, and does the answer differ between a small classifier and a fine-tuned encoder? The decision-head paper left that dose open. It also left the deliberate mode unmeasured: the decision head ships a mode that asks each judgement question eight times at temperature 0.7 and takes a vote, and nobody had checked whether it earns its cost.

The registration also proposed a property-claims process and a head that reads the whole case file. The first adversarial review moved claims to J10, since that family needs its own scorer and export, and found that the compact state already carries every document verbatim, so a full-file arm had nothing to add. Both were withdrawn before any build (Addendum 1).

## 2Design as registered

Process and splits. Supplier onboarding, the process with the most banked reference arms. A new training split, onboarding-train-big, holds 3,000 cases (seed 35), generated as thirty independent 100-case plan blocks so that every block carries the evaluation split's mix of decoy, near-boundary, routine and hard cases. Evaluation uses the 200-case onboarding challenge split and the seed-17 200-case subsample of eval2, so every curve point shares cases with the published numbers of the three earlier heads. The split and the draws were hashed into the manifest on 3 October.

Draws. Six draws of whole blocks, sampled without replacement in one pass and so pairwise disjoint: draw 201 (100 cases), 203 (300), 205 (1,000), 206 (3,000, the whole split), and second draws 202 (100) and 204 (300). The second draws give the curve's empirical noise floor. No claim rests on a gap smaller than the spread between a first and second draw at the same size.

Three arms.

- L@n, the learned head with L0 features, trained on the judging head's run over the draw's cases. It serves without any call to the judging head.
- LY@n, the open encoder fine-tuned through the J8 export unchanged, seed 0, same hyperparameters at every size.
- C‴ (Cprime3d), the deliberate mode exactly as shipped: k = 8, temperature 0.7, reason-then-answer, judgement question only, vote share thresholded at the head's T = 0.7, which means six of eight. It runs on onboarding challenge and accounts-payable challenge beside a fresh C″ control in the same window. A malformed sample counts as a "no" vote, and a malformed share above 5 % stops the arm.

Banked references: the rule table A, the judging head C″ and the untuned encoder LY-0.

The walk ceiling. Before any J9 call, a replay script replayed C″'s elimination walk with an oracle judge on banked states. With a perfect judge the walk reaches 0.822 on challenge and 0.795 on the registered 200-case subsample; with judgement switched off, 0.499 and 0.490. A learned head inside this walk can pass the ceiling only by the small margin its own trajectory moves it; the accuracy that remains is lost in the walk's structure. Each arm is read against the ceiling recomputed on its own states (primary) and against the banked line (printed beside it).

The lexical probe. A four-feature rule on the observation alone, fit on the J5 training run, used as the walk's judge, reaches 0.754 on the 200-case subsample and 0.793 on challenge. It is a line on every chart. If an arm sits within 0.03 of the probe at every size, the paper must say the curve measures how fast a head picks up a cue the observation already carries.

Readings, fixed in Addenda 3 to 5. "Within ±0.03" means the point estimate of a difference lies in [−0.03, +0.03]; two arms are called equivalent only when the 95 % case-clustered interval lies inside ±0.06. "Reaches the ceiling" means the 95 % interval of (ceiling − arm) lies entirely below 0.05. Every interval is a paired, case-clustered bootstrap, 10,000 resamples, seed 0.

Hypotheses. H30′: L@100 below A with its interval clear of zero; L@300 within ±0.03 of A; L@1,000 reaches the ceiling; L@3,000 − L@1,000 within ±0.03; falsified by an adjacent decrease clear of zero or a top step above +0.05. H31′: LY@100 below A; LY@300 reaches the ceiling, as J8 did at 363 contributing cases; LY@3,000 − LY@300 within ±0.03; falsified if LY@100 ≥ A or if the ceiling is first reached at 1,000 or later. H33′: C‴ − C″(fresh) ≥ +0.03 with the interval above zero on both splits, predicted +0.04 (band +0.01 to +0.08); below +0.01 on both withdraws the product claim.

Kill conditions. Transport failures above 2 % stop a launcher. Any arm whose draw has fewer than 0.5 × n cases contributing training rows stops. Fine-tunes stop at 3 hours, 5 hours for the 3,000 draw. The deliberate arm stops above 5 % malformed samples once 50 cases of a split are in.

The cut. Priority order: train-big and the single-draw L curve, then the single-draw LY curve, then C‴ with its controls, then the second draws. Anything not started by the cut instant is cut and recorded. Addendum 6 moved the instant from 12 to 14 October 12:00 Chicago when the launch slipped from 8 to 11 October, keeping the same 62 hours of slack, before any J9 data existed.

## 3What happened in the run

The 8 October launch did not happen because the laptop that hosts the launcher was away. J9 launched on 11 October, ahead of another study's compute window.

It took three starts. The first stopped in preflight at 09:24 on a shell glob in the mini's lock check that errors under zsh; the check was rewritten with the same meaning. The second stopped at 09:28 before phase 1a because an unrelated daemon held port 8787, and the fast head moved to 8788 through its documented setting. The third, at 09:29, passed the five-case smoke and began writing train-big episodes at 09:32. Nothing had touched the endpoint or a run directory before it.

Phase 1a, the judging head over all 3,000 training cases, ran for about 25 hours at four streams against a budget line of 20.8 hours, and finished at 10:26 on 12 October with zero transport failures in 3,000. The endpoint served the lab's assistants throughout, as registered.

A power guard went on at 17:24 on the first day, after the worker GPU host read a maximum of 191 W at the wall against an earlier power-off at about 200 W. It stops the launcher, resumably, after three consecutive samples at or above 190 W on the worker or 210 W on the head, and changed no run rule. It never stopped the launcher, and no stop file was written.

Phase 1b stopped once. Draw 201 passed its dose check, 60 of 100 cases contributing rows against the 0.5 floor, and then the launcher's call to the trace writer passed its two arguments in the wrong order and raised a TypeError before any file was written. The call was corrected and the launcher resumed at 10:26, skipping train-big as banked. Phase 1 closed at about 19:40 on 12 October: eight L cells, four sizes on each of two evaluation sets, and zero transport failures in 4,600 episodes.

Phase 2a wrote the four encoder exports at 19:36: 705 rows for draw 201, 2,397 for 203, 8,017 for 205 and 23,448 for 206. The fine-tunes ran back to back on the laptop's GPU: 8 minutes for 100 cases, 33 for 300, 116 for 1,000, all clean. The 3,000-case fine-tune, 11,724 steps at about 1.4 seconds, hung at step 6,800 inside the GPU graph compiler, with no log line after 01:16 on 13 October and the GPU idle. Its projected 337 minutes already exceeded the 5-hour kill line, so the cell was lost either way. The process was killed at 02:25 and the cut log records LY@3,000 as cut.

The deliberate arm's malformed share was 3.7 % of samples on challenge and 2.5 % on accounts payable, under the 5 % stop line, so it was scored. The cut log holds a single line: LY@3,000, recorded at 09:32:36 on 13 October, fine-tune failed. The launcher finished at 07:09:25 on 14 October, before the 12:00 cut instant, so the mechanical cut removed nothing else and both second draws ran. In all, 8,200 episodes ran with zero transport failures: 3,000 training cases and 26 evaluations of 200.

## 4Results

The analysis prints each quantity per evaluation set. Rows that depend on the cut LY@3,000 cell print NO DATA by construction, and the script's "never reached by 3,000" and "CUE-ONLY at every n" summaries read NO DATA for the encoder for the same reason.

***Figure 1. Decision-class accuracy of the from-scratch classifier (L) and the fine-tuned encoder (LY) by draw size, on both evaluation splits, against the walk ceiling with a perfect judge, the lexical-probe walk and the rule table. Hollow points are the second draws at 100 and 300 cases. Values from the recompute's cell table, which matches the result files in scored and correct counts.*

| arm | challenge (200) | eval2 subsample (200) | contributing cases | decisions with open items |
| --- | --- | --- | --- | --- |
| A (rules) | 0.565 | 0.553 |  |  |
| C″ (banked) | 0.739 | 0.732 |  |  |
| L@100 | 0.397 (0.458) | 0.408 (0.531) | 60 (61) | 247 (287) |
| L@300 | 0.501 (0.509) | 0.544 (0.532) | 180 (180) | 832 (810) |
| L@1,000 | 0.463 | 0.508 | 606 | 2,791 |
| L@3,000 | 0.526 | 0.523 | 1,821 | 8,151 |
| LY@100 | 0.642 (0.711) | 0.634 (0.675) | 60 (61) | 247 (287) |
| LY@300 | 0.713 (0.807) | 0.712 (0.755) | 180 (180) | 832 (810) |
| LY@1,000 | 0.825 | 0.798 | 606 | 2,791 |
| LY@3,000 | cut | cut |  |  |
| walk, oracle judge (banked states) | 0.822 | 0.795 |  |  |
| walk, lexical-probe judge | 0.793 | 0.754 |  |  |

Table 1. Decision-class accuracy by arm and dose, primary draw (second draw in brackets), from the recompute, whose 22 cell accuracies match the result files in scored and correct counts. Contributing cases and decisions with open items from the dose records.

### The learned classifier, H30′

| claim | registered threshold | predicted | challenge | eval2 subsample | verdict (challenge / eval2) |
| a. L@100 − A | interval below zero | −0.05 | −0.167 [−0.235, −0.092] | −0.144 [−0.221, −0.064] | SUPPORTED / SUPPORTED |
| b. L@300 − A | point in ±0.03 | about 0 | −0.063 [−0.133, +0.010] | −0.008 [−0.072, +0.056] | NOT SUPPORTED / SUPPORTED |
| c. L@1,000 reaches ceiling (own states) | interval of gap < 0.05 | reaches | +0.265 [+0.188, +0.342] | +0.206 [+0.139, +0.278] | NOT SUPPORTED / NOT SUPPORTED |
| d. L@3,000 − L@1,000 | point in ±0.03 | ≤ +0.03 | +0.062 [+0.000, +0.125] | +0.015 [−0.028, +0.058] | NOT SUPPORTED / SUPPORTED [EQUIVALENT] |
| falsifier: adjacent decrease | interval below zero | not observed |  |  | not observed / not observed |
| falsifier: top step | point > +0.05 | not observed |  |  | FALSIFIED / not observed |
| ceiling never reached by 3,000 | no size reaches | FALSE |  |  | TRUE / TRUE |
| probe: CUE-ONLY at every n | all four within 0.03 of probe |  |  |  | not at every n / not at every n |
| noise floor, n = 100 | reported |  | +0.061 [−0.034, +0.155] | +0.123 [+0.031, +0.215] |  |
| noise floor, n = 300 | reported |  | +0.007 [−0.060, +0.073] | −0.013 [−0.071, +0.044] |  |

Table 2. H30′, the learned classifier, from the registered analysis output.

The from-scratch classifier's own-state gap stays between +0.183 and +0.396 at every size on both splits, so no size reaches the ceiling, and it sits 0.147 to 0.374 below the probe walk on its own states, so no size reads as cue-only. At 300 it is level with the rules on eval2 (−0.008) and below them on challenge (−0.063). From 300 to 3,000 cases it moves from 0.501 to 0.526 on challenge and from 0.544 to 0.523 on eval2. L@1,000 dips below L@300 on both splits, a dip the script does not flag. On challenge the top step, L@3,000 − L@1,000 = +0.062, exceeds the +0.05 falsifier, which therefore reads FALSIFIED; that step is the same size as the 100-case draw spread on the same split (+0.061). On eval2 the top step is flat and equivalent. Per size, the own-state ceiling gaps on challenge are +0.396 [+0.302, +0.487], +0.232 [+0.162, +0.308], +0.265 [+0.188, +0.342] and +0.262 [+0.188, +0.336], and the distances to the lexical probe are −0.374 [−0.467, −0.277], −0.217 [−0.294, −0.144], −0.251 [−0.332, −0.170] and −0.242 [−0.318, −0.166].

***Figure 2. Ceiling on each arm's own states minus the arm's accuracy, with the 95 % case-clustered interval. The registered reading "reaches the ceiling" needs the whole interval below 0.05 (shaded). Only LY@1,000 qualifies, on both splits. From the registered analysis output, lines 2 to 8, 20 to 24, 36 to 42 and 54 to 58.*

### The fine-tuned encoder, H31′

| claim | registered threshold | predicted | challenge | eval2 subsample | verdict (challenge / eval2) |
| a. LY@100 − A | interval below zero; falsified if point ≥ 0 | below A | +0.077 [−0.017, +0.171] | +0.082 [−0.008, +0.171] | FALSIFIED / FALSIFIED |
| b. LY@300 reaches ceiling (own states) | interval of gap < 0.05; falsified if first reached at 1,000 | reaches (J8: 363 cases) | +0.095 [+0.043, +0.155] | +0.086 [+0.038, +0.144] | FALSIFIED / FALSIFIED (ceiling first reached at 1,000 or later) |
| c. LY@3,000 − LY@300 | point in ±0.03 | flat | cut | cut | NO DATA (cell cut) |
| falsifier: LY@3,000 − LY@1,000 | point > +0.05 | not observed | cut | cut | NO DATA (cell cut) |
| LY@1,000 own-state gap | reported |  | +0.005 [+0.000, +0.017] | +0.000 [+0.000, +0.000] | SUPPORTED / SUPPORTED |
| probe distance, LY@100 / 300 / 1,000 | CUE-ONLY if within 0.03 |  | −0.087 [−0.154, −0.026]; −0.070 [−0.135, −0.013]; +0.016 [−0.005, +0.035] | −0.037 [−0.085, +0.001]; −0.047 [−0.109, +0.005]; +0.035 [+0.022, +0.049] |  |
| noise floor, n = 100 | reported |  | +0.069 [+0.010, +0.132] | +0.041 [−0.042, +0.122] |  |
| noise floor, n = 300 | reported |  | +0.094 [+0.045, +0.149] | +0.043 [−0.020, +0.103] |  |

Table 3. H31′, the fine-tuned encoder, from the registered analysis output.

LY@1,000 sits +0.005 and +0.000 below the ceiling on its own states and 0.002 above the banked reference line on both splits, which puts the encoder at the ceiling with 1,000 cases. LY@300 misses by +0.095 and +0.086. The second 300-case draw scored 0.807 and 0.755, and the challenge spread at 300 (+0.094) equals the challenge gap the first draw would have to close, so where the crossing falls between 300 and 1,000 rests on one draw per size. LY@100 already beats the rules on point estimate on both splits (+0.077, +0.082), which falsifies claim a. That dose, 247 decisions from 60 contributing cases, is close to the 201 decisions that left J8's accounts-payable encoder below the rule table.

***Figure 3. The curve's noise floor: second draw minus first draw at 100 and 300 cases, with 95 % intervals and the registered ±0.03 margin shaded. At 300 cases on challenge the encoder's spread, +0.094, is as large as the gap its first draw left to the ceiling. From the registered analysis output, lines 18, 19, 34, 35, 52, 53, 68 and 69.*

The probe rows carry a flag. On challenge, LY@1,000 sits +0.016 [−0.005, +0.035] above the lexical-probe walk on the same states, inside the registered 0.03, and the script marks it CUE-ONLY. On eval2 it sits +0.035 [+0.022, +0.049] above, outside the margin. The registered consequence needs the flag at every size. Its summary row reads NO DATA because LY@3,000 is missing, and LY@100 and LY@300 sit 0.087 and 0.070 below the probe on challenge, so the condition fails at those sizes in any case. The flag means that a four-feature rule on the observation, walked through the same states, scores what the encoder scores on challenge, so that split cannot separate judgement from cue at 1,000 cases. Whether the encoder uses those features is a separate question the flag leaves open.

The cut changes what H31′ can say. Claim b is still falsified, because the script checks whether 1,000 reached the ceiling while 300 did not. Claim c, the flat top, has no reading, and the open question it leaves is whether more labels hold the encoder at the ceiling it already reaches at 1,000.

### The deliberate mode, H33′

| split | registered threshold | predicted (band) | C‴ − C″ fresh | C″ fresh − C″ banked (rerun noise) | verdict |
| onboarding challenge | ≥ +0.03, lower bound > 0 | +0.04 (+0.01 to +0.08) | −0.010 [−0.039, +0.022] | −0.002 [−0.025, +0.019] | does not support |
| accounts-payable challenge | same | same | +0.008 [−0.027, +0.045] | +0.021 [−0.010, +0.053] | does not support |
| overall | both support; WITHDRAWN if both < +0.01 | SUPPORTED |  |  | WITHDRAWN |

Table 4. H33′, the deliberate mode against a fresh control, from the registered analysis output, lines 70 to 74.

Cost, measured: 44.08 model calls and 184 seconds per case for C‴ on challenge, 38.08 calls and 160 seconds on accounts payable, against 20.3 and 16.1 seconds per case for the fresh C″. Both differences sit inside the rerun noise printed beside them, the gap between two runs of the same single-call head on the same cases. Both points fall below +0.01, the registered withdrawal line, and the predicted +0.04 lies outside the challenge interval and inside the accounts-payable one, which runs to +0.045.

***Figure 4. H33′. Eight sampled votes minus one call on each split, beside the difference between a fresh single-call run and the banked one on the same cases. The predicted band is shaded; the registered support line is +0.03 and the withdrawal line +0.01. Both points sit below the withdrawal line; the predicted +0.04 (vertical line) falls inside the accounts-payable interval.*

Against the earlier heads. On the fresh eval2 split the judging head scored 0.728 against the rule table's 0.566 (n = 800). On challenge, J8's fine-tuned encoder, trained on 363 contributing cases out of a 600-case split, scored 0.815 against C″'s 0.739 and the rules' 0.565, within 0.01 of the walk ceiling. J9's LY@300, at 180 contributing cases, scores 0.713 on challenge, short of J8's figure; its second draw, at 0.807, comes within 0.01 of it, and LY@1,000, at 606 contributing cases, reaches 0.825. J9 places the crossing between 180 and 606 contributing cases, on one draw at each end.

***Figure 5. Cost to serve and cost to train. Left, decision-head seconds per case on a log scale: L 0.015 to 0.02 s and LY 0.35 to 0.51 s from the analysis notes, C″ fresh 16.1 and 20.3 s, C‴ 160 and 184 s. Right, encoder fine-tune wall time per draw from Addendum 6, with the 3,000-case run's projected 337 minutes against the registered 5-hour line.*

## 5Comparison with J9-C

J9-C, registered on 11 October and run the same morning, asked J9's two questions with a different mechanism. A small-tier and a mid-tier general model read labelled examples in context, with no fine-tuning, and classified one checkpoint per case on the same seed-17 eval2 subsample. Its labels came from draw 201, the same frozen draw J9's 100-case arms trained on.

The in-context curve was steep and short. The mid tier went from 0.725 with no examples to 0.840 with ten and 0.890 with a hundred; the small tier gained 0.09 by ten examples and nothing after. The gain sat in BLOCKED, where recall rose from 0.02 to 0.80 for the mid tier at a hundred examples. J9-C's lead read this as the examples teaching the generator's conventions: a request already made means hand off, an unrelated-party document on file means unresolvable. That is the cue the lexical probe encodes.

The two studies do not share a scale. J9-C scores one checkpoint per case against the oracle class; J9 scores every decision in a live episode inside a walk whose ceiling is 0.795 on this subsample, so a J9-C figure above 0.795 is no evidence that in-context labels beat the walk. Shape can be compared. The in-context gain came from BLOCKED. The fine-tuned encoder reaches the walk ceiling at 1,000 cases on REASON and INFO: on challenge LY@1,000 recalls 0.939 of REASON decisions, 0.973 of INFO and 0.301 of BLOCKED. The walk itself caps BLOCKED, since with an oracle judge its BLOCKED recall is 0.32 on challenge and 0.30 on eval2, so the walk leaves a J9 head no BLOCKED headroom above those figures. The trained encoder also climbs over a longer range, 0.642 at 100 cases to 0.825 at 1,000 on challenge.

On the deliberate mode, J9-C's eight calls agreed on 198 of 200 checkpoints and gained nothing, but repeated agent calls with one prompt are a different thing from a temperature sampler, and its write-up says so. J9's C‴ is the sampler as shipped. Its result, WITHDRAWN, is the first measurement of what k = 8 voting buys a logprob head.

## 5aAddendum, 16 October: Kev, an untrained open decision model, as the head

After J9 closed, a fourth head ran on the same case sets under a registration written before any case ran (PREREG-K, 15 October). Kev is an open decision model of the small tier, a low-rank adapter and pointer head on a general base, released in September, with no training on this lab's data, served on a laptop and reached through the same decision API as the hosted reference. Nothing in the harness changed; only the source of the six probabilities did.

| split | rules | Kev, untrained | large judging head (fresh) | fine-tuned encoder at 1,000 | ceiling |
| challenge | 0.565 | 0.651 [0.603, 0.699] | 0.738 | 0.825 | 0.822 |
| eval2 subsample | 0.553 | 0.601 [0.550, 0.650] | n/a | 0.798 | 0.795 |
| accounts-payable challenge | n/a | 0.669 [0.623, 0.716] | 0.681 | n/a | n/a |

Table 5. Kev against the references, from the independent recompute of the K runs. Reference points are that recompute's rescore of the banked runs; the rules read 0.565 and 0.553 there, where the registration quoted 0.564 and 0.552.

***Figure 6. Kev in the head's seat, same walk and same cases, against the rule table, the fresh judging head, the encoder at 1,000 cases and the walk ceiling. Kev's 95 % interval is a case-level bootstrap, 10,000 resamples, seed 0.*

Every reading here was registered as a number before the run. Kev beats the rules on both onboarding splits, +0.086 [+0.041, +0.134] and +0.048 [+0.017, +0.082]. It lands at 0.651 on challenge, inside the predicted band of 0.55 to 0.71 and 0.021 above the point prediction of 0.63, and 0.088 [0.031, 0.144] below the fresh judging head. It trails the fine-tuned encoder by 0.175 [0.130, 0.221] and 0.197 [0.144, 0.250], both intervals clear of a 0.10 gap, so the 1,000 labelled cases bought something an off-the-shelf decision model does not ship with. The registered falsifier would have fired at 0.78 on challenge, and Kev's upper interval end is 0.699. On the accounts-payable family it ties the large judging head, −0.012 [−0.057, +0.033], inside ±0.08, which says the question map carries that family and the model behind it matters less.

Two secondary readings came out off their predictions. The confident-error share was 0.000 on every split, below the predicted band of 1 to 10 %: only 14 decisions in total reached a confidence of 0.90, and all were class-correct. Kev's shipped temperature flattens its probabilities, so that threshold rarely binds. Its wrong-claim rate on challenge was 0.235 against 0.355 for the fresh judging head on the same cases, and 0.005 on the eval2 subsample; it handed off 80 of the 200 challenge cases, where the judging head handed off 58. All Kev numbers were recomputed independently from the run directories and matched the scorer to full precision, except one definitional difference in the accounts-payable wrong-claim rate (0.305 scorer, 0.300 recompute, one budget-exhausted case).

## 6Limits

Sizes 1,000 and 3,000 each have a single draw, and 3,000 is the whole split. The spread between draws at 100 and 300 is the only noise estimate the curve has, so it is extrapolated upward, and at 300 cases on challenge it equals the encoder's gap to the ceiling.

Every model call in the run goes to one open-weight model family on the lab's endpoint, which leaves the family itself unnamed and untested. A different family could move the ceiling's position and C‴'s gain.

The scorer's DONE and BLOCKED classes are the ones model judges read differently from the rule table. The Mirror study found three judge tiers disagreeing with the oracle in the same places, and J9-C's adjudicator never chose BLOCKED, so every accuracy here is agreement with a programmatic oracle on a synthetic process. Where a J9 figure sits beside a judged figure, such as J9-C's adjudication or the Mirror's agreement rates, read about ±0.05 of judge noise on top of the sampling interval; that is the Mirror's convention, half the observed test-retest disagreement of 19 in 197. J9's own scoring carries no judge.

Encoder fine-tunes ran on a laptop GPU, at 500, 1,981 and 6,945 seconds for draws 201, 203 and 205. On that host the 3,000-case run would not fit the 5-hour line, and it hung in the graph compiler, the second such stall this harness has seen. A hardware fault removed the encoder's top dose, and the flatness of its curve above 1,000 cases is unmeasured.

The cut log holds one cell, LY@3,000, recorded at 09:32:36 on 13 October after its fine-tune failed. Every other registered cell ran before the 14 October cut instant. H31′ c, its falsifier and the encoder's probe consequence depend on the cut cell and have no reading.

## 7Scope of the claims

Every number belongs to one synthetic onboarding process; transfer to accounts payable, claims or a real process is untested, and the claims family is the next registration, J10. The deliberate mode's cost was measured at k = 8 and temperature 0.7 on this endpoint and is read only there. J9 and J9-C sit on different scales, and this paper compares them by shape alone.

## 8Reproducibility

Registration: the J9 pre-registration, Addenda 1 to 6, the launch record and Addendum 7, the closing record. Analysis: the registered analysis script, unchanged since Addendum 5, selftests 8 of 8 at launch, run at 07:10:31 on 14 October. The walk ceiling, the lexical probe, the dose results, the fine-tune and cut logs, a sample of the hung fine-tune and the split hash are kept with the run records. Code and data are available on request until the public repository opens.

Recompute: done at 07:17 on 14 October by an independent agent from the raw run directories with its own code (seed 20261014, 10,000 resamples), opening the analysis output only after writing its own numbers. All 22 cell accuracies are identical in scored and correct counts. Of 55 difference rows, 36 match at three decimals with interval ends within 0.003; the other 19, the reference-line gaps and C″fresh − C″banked on challenge, differ by at most 0.007 because the analysis uses the cases both sides have rows for (199 on challenge, 197 on eval2), and match to three decimals on those cases. The lexical-probe rows were not recomputed, because the probe script runs only on banked states.

LY is the Laya open-encoder recipe, fine-tuned through the J8 export; the draw-205 checkpoint holds 421,293,830 parameters, 394,781,696 of them in the encoder. Kev, the untrained head of section 5a: the K pre-registration with Addenda 1 to 3, its run directories and scores, and an independent recompute (own code, paired case-level bootstrap, 10,000 resamples, seed 0).

Workflow-completion campaign, registration J9, with the K addendum.

Continue the thread

## A next room to enter.

[Choose another path ](https://dinand.com/)

[Research · Study summary

### The Judge’s Mirror

How often a model judge agrees with itself on 197 blind case files, and how far down one model family a judge stays usable.

](https://dinand.com/research/the-judges-mirror/)[LinkedIn · 2026

### Skin in the Game Made the AI Supervisor Worse

A controlled experiment on the cost of wrongful blocks and the incentives that shape an AI reviewer.

](https://dinand.com/publications/skin-in-the-game/)[Medium · 2026

### Inventory Optimization Under Uncertainty: Broad random shocks trained a tougher inventory planner than a clever AI adversary did

An inventory-planning experiment explores which kinds of stress prepare a policy for unfamiliar disruptions.

](https://dinand.com/publications/inventory-under-uncertainty/)

[All Research ](https://dinand.com/research/)[Back home ](https://dinand.com/)

---
Site guide for AI agents: https://dinand.com/llms.txt