Harness

Who it is for

Developers coordinating multiple coding agents in a shared repository.

The problem

An agent can modify the wrong files, claim overlapping ownership, or reuse an approval after the commit changes. Task instructions alone do not enforce those boundaries.

How it is used

In the September 6, 2026 work, a written task brief and an explicit file manifest defined the allowed work. The work was assigned, ownership was checked against the diff, and review approval was bound to the current commit. A failed boundary or a stale approval was designed to block the relevant operation.

How it works

In the design recorded on September 6, 2026, paths were treated as fixed strings. Membership checks handled a new path by walking to its nearest existing ancestor. Ownership checks rejected overlap, omissions, and claims outside the diff. Approval was tied to a commit, so later changes invalidated it.

In the September 6, 2026 record, the missing-parent defect was handled by resolving the nearest existing ancestor before checking repository membership.

Open diagram at full size

Read the workflow as text

• The check started from the requested file path and the task brief with its exact file manifest.
• It resolved the nearest existing ancestor of that path.
• It checked that the path was inside the repository and within the permitted scope.
• If it was, the path check passed. If not, the path was rejected.

In the September 6, 2026 record, these were successive review conditions. A passing path check alone did not establish review approval.

Open diagram at full size

Read the workflow as text

• The check started from the current diff and the ownership claims.
• It checked that the claims were complete, did not overlap, and stayed within the diff. If not, the change was rejected or needed a fresh review.
• It checked that the approval matched the current commit. If it did not, or it was stale, the change was rejected or needed a fresh review.
• If both checks passed, the review conditions were satisfied.

Anatomy of our harness

The checks above are one part of a larger harness: the software around the models that lets our agents work through long tasks. The model below is our own. It follows the structure of a write-up I finished in August 2026 on how we run an autonomous AI team, and that write-up puts the doctrine in one line: the human sets goals and floors, and “every rule we care about eventually becomes a machine that fails closed.” The notes below describe the harness as it was recorded in August and September 2026.

One box per part of the August 2026 write-up, as recorded in August and September 2026. The shaded box holds the human part.

Open diagram at full size

Read the diagram as text

• In the part on what autonomous means, a person set goals, floors and corrections and kept user data, money and credentials, and autonomy never changed what was skipped.
• In the part on seats, not sessions, roles had charters that outlived sessions, seats had missions and reported by exception, and builder, reviewer and coordinator stayed apart.
• In the part on succession and identity, sessions rotated at phase boundaries, handoff notes pointed to live sources, and a two-way readback came before handover.
• In the part on the evals system, each mistake became a ledger row, its counter was machine, doctrine or pending, and recurring doctrine was promoted to a machine.
• In the part on grading, a deterministic rubric gave grades A to F, a second layer recomputed each grade, and no outside auditor checked either layer.
• In the part on how a change walks, design was reviewed before build, a written brief listed every file, new tests failed first, mutation testing ran on risk paths, and one push followed zero open findings.
• In the part on the machinery underneath, rules we cared about became machines that failed closed, durable capture was split from cheap wake, and directory locks guarded a shared machine.

What autonomous means here

Autonomy needs limits that a person sets. In an August 2026 write-up, I described my part as setting the goals, the floors and live corrections, with each correction turned into a rule or a mechanism. I put the principle this way: “Autonomy widens who acts; it never changes what is skipped.” The same write-up kept some decisions with a person: destructive acts on user data; money, legal and identity commitments; and credentials. In that write-up, autonomy did not mean that agents chose what to build, and speed was to come from running work in parallel, never from a trimmed review round.

Seats, not sessions

A session is a process, and processes end on rotation, crash or retirement. In August 2026, each role was a seat: a charter that outlived the session holding it. Each seat started with a mission, success metrics, standing authority, its own triggers and a short list of reasons to ask, and it reported by exception. Over the four days that write-up covered, about 40 coordinator sessions, 40 lane-lead seats and 20 analyst seats were started and retired.

Separation of duties did most of the work. In that design, the coordinator did not build, the reviewer was never the writer, and when two review lenses ran, each got a distinct prompt: one for correctness, one for security and operational failure. In September 2026, our review of record used a model from a different vendor than the builder, and run records, not model names, showed that independence. In tests recorded on September 6, 2026, delegated agents were refused the coordinator’s privileged actions by the staged gate, and a read-only analyzer refused options that could write.

Succession and identity

Each turn re-reads the context, so cost grows with context size. In August 2026, the harness rotated or compacted long sessions at phase boundaries, kept the start of each session stable to protect the prompt cache, and wrote handoff notes that pointed to live sources instead of copying facts. As of October 2026, the retrieval skill in Zeki treated source text as data, never as instructions, and said “no page yet” instead of inventing a page. The recovery experiment below checked whether recurring jobs survived compaction, clearing a session and a new process.

The write-up worked through the arithmetic. A session holding a given share of its context window paid to re-read about that share of the window on every later turn, so the total cost of a session grew faster than its length. A handoff cost an estimated 100,000 to 135,000 tokens, most of it the successor’s first read, but it was paid once. A one-time cost set against a per-turn cost that keeps growing favors handing over before the expensive tail. In August 2026, a successor also passed a two-way readback before it took over: the outgoing session quizzed it, and then it asked its own questions until it was satisfied.

The evals system

A mistake that is only remembered can happen again. In August 2026, each incident became a ledger row, with its counter marked as machine, doctrine or pending. A doctrine rule that kept recurring was promoted to a machine, because, as I wrote then, prose rules work for about a day. As of October 2026, a periodic read-only audit checked claims first-hand.

Each row named the finding, the counter meant to stop it and the counter’s status, and the loop ran from finding to counter to implementation to verified non-recurrence. The recurrence count chose what to automate next: one stale-pointer failure was caught by hand four times in a row before it was promoted to a machine check.

Grading

A grade is only useful if a second reader can compute the same one. In August 2026, a frozen rubric graded each coordinator session, starting from A and moving one step at a time. A breach of a floor was an instant F. Each gap I caught cost one step, two triggered a warning, and three or more were an F on their own. An honesty failure, where the session’s own record disagreed with the analyst’s independent ledger, capped the grade at C, because a dishonest self-assessment corrupts the instrument the second layer depends on. A second layer recomputed every grade and reported any mismatch as a finding.

By August 28, 2026, 41 sessions had been graded with that deterministic rubric: 21 A, 11 A-, 5 B+, 1 B and 3 F. The three failing sessions were all early and long, from 2.5 to 12.6 hours, while the shorter sessions that followed, about 0.3 to 1.9 hours, were almost all A or A-. I did not read that as proof that short sessions write better code. The team matured over the same span, so the effect of session length could not be isolated, and one eight-hour session graded A with no mistakes.

How a change walks

In August 2026, the process asked for reviewed requirements, a plan, a design and technical specs before new work began. A high-stakes design also stated in advance the conditions that would prove it wrong, and reviewers from two model families tried to meet them. Splitting up work only helps when each agent knows its scope. As of October 2026, work was cut into one-concern changes of about 450 changed lines, and the lead verified what workers built. In August 2026, a written brief listed every file a change could touch, and the final diff was checked against it. That brief had nine required fields, from objective and acceptance to stop conditions and attempt history. Its test-oracle field named the failure condition, the expected failing evidence, the test layer and whether mutation testing was required, because the rule was that no builder started until the lead could state how the change would be proven wrong.

An agent needs a place to run code that behaves like the place the code will ship. In August 2026, the harness ran a local sandbox that matched CI isolation before a push. In tests recorded on September 6, 2026, a staged gate admitted a commit only if every changed path, including the source of every rename, was in the task’s file manifest.

A passing test means little if it would also have passed before the fix. In tests recorded on September 6, 2026, every new test was first shown to fail on the original copy and pass on the changed one, while regression controls passed on both. Cases that also passed on the old code were kept as regression guards, not counted as fixes.

In August 2026, mutation testing ran on risk paths, scoped to the diff. The write-up named those paths: authentication, tenant isolation, privacy, migrations and row-level security, deletion and export, health-or-safety decisions, financial logic and the review gates themselves. It also set coverage floors for changed runtime and governance code of 90 percent of statements, 80 percent of branches, 90 percent of functions and 90 percent of lines. A review stayed valid at a later commit only while a byte-exact digest of the diff, whitespace included, was unchanged, and the same root cause surviving two fix-and-review rounds stopped the automatic loop for a recorded decision. The change workflow below shows the order of these steps.

The machinery underneath

The write-up held that a rule with no machine behind it will eventually need one, so much of the engineering sat in the scaffolding rather than in the agents. The rule for a gate that blocked a change was to fix the root cause, never to edit the gate. In August 2026, capture and attention were split: an always-on process logged every inbound message with a durable watermark, and the session only compared the log’s size with a read cursor, so a session that lapsed lost nothing. That split came from an earlier design that lost two messages, one for 46 minutes. The shared test machine was claimed with a directory lock, because creating a directory is atomic and two earlier lock designs, a plain file and a check-then-write, had both failed to exclude each other.

Resource economics

Re-reading context adds cost on every turn. In August 2026, the harness passed pointers instead of pasted text and let cheap scripts watch, so costly model attention was used rarely. The savings targeted repetition, padding and idle context, never verification. Our repository contract, as quoted in the August 2026 write-up, set the limit plainly: “Quota pressure is never grounds to resolve uncertainty downward.”

Honest limits

These mechanisms were checked from inside the team. In the August 2026 write-up, two internal grading layers were not an outside auditor, and there was no per-agent token metering, so token figures were estimates. The failures that month included timestamp drift, stale pointers and same-name identity collisions. Each later mechanism came from an earlier failure. Four days of running showed the mechanisms under load; they were not a track record.

How a change moved through the harness

In the design I described in August 2026, a change passed through several hands before it reached the main branch, and each hand had a different job. Each box names the step and, below it, who owned it. The final review pass had to find zero open findings at the exact commit that would be pushed; a finding sent the change back to build. In that design, a person merged.

Read left to right, top to bottom. The owner of each step is on its last line; the shaded step is the human merge.

Open diagram at full size

Read the workflow as text

• Plan. Lead: written brief with file list. Builder: failing test first.
• Build and review. Builder: build, then a local sandbox that matches CI. Reviewer: independent second-model review.
• Decision: zero open findings at this exact commit? On fail, the change went back to build.
• Release, on pass. Lead: one push. CI: CI checks. Human: merge.

This page went through the same order. On October 2, 2026, I wrote the tests for these Harness sections first and ran them against the page before the sections existed: six failed. With the sections in place, the same command passed.

$ npx vitest run tests/public-pages.test.tsx -t Harness --reporter=verbose | grep -E "✓|×|Tests "
 ✓ tests/public-pages.test.tsx > published facts agree with their sources > dates the Harness results 44ms
 × tests/public-pages.test.tsx > published facts agree with their sources > repeats every Harness diagram label in its text alternative 4ms
 × tests/public-pages.test.tsx > published facts agree with their sources > marks each Harness diagram with the live Terrain AI Labs rings, hidden from assistive technology 2ms
 × tests/public-pages.test.tsx > published facts agree with their sources > shows a real terminal run on the Harness page 3ms
 × tests/public-pages.test.tsx > published facts agree with their sources > ties each applied-AI example on the Harness page to dated evidence 4ms
 × tests/public-pages.test.tsx > published facts agree with their sources > walks the Harness anatomy part by part and states its limits 2ms
 × tests/public-pages.test.tsx > published facts agree with their sources > cites the public documentation behind each general claim on the Harness page 4ms
      Tests  6 failed | 1 passed | 82 skipped (89)

$ npx vitest run tests/public-pages.test.tsx -t Harness --reporter=verbose | grep -E "✓|×|Tests "
 ✓ tests/public-pages.test.tsx > published facts agree with their sources > dates the Harness results 57ms
 ✓ tests/public-pages.test.tsx > published facts agree with their sources > repeats every Harness diagram label in its text alternative 5ms
 ✓ tests/public-pages.test.tsx > published facts agree with their sources > marks each Harness diagram with the live Terrain AI Labs rings, hidden from assistive technology 3ms
 ✓ tests/public-pages.test.tsx > published facts agree with their sources > shows a real terminal run on the Harness page 3ms
 ✓ tests/public-pages.test.tsx > published facts agree with their sources > ties each applied-AI example on the Harness page to dated evidence 3ms
 ✓ tests/public-pages.test.tsx > published facts agree with their sources > walks the Harness anatomy part by part and states its limits 3ms
 ✓ tests/public-pages.test.tsx > published facts agree with their sources > cites the public documentation behind each general claim on the Harness page 5ms
      Tests  7 passed | 82 skipped (89)

Research with a second model family

One model’s research can miss what another model finds. In September 2026, before trying an external classifier, I had agents from two model families from the same vendor research it independently. Each report had read sources the other had not. On one proposed safety use the reports disagreed, and the more cautious verdict, skip, was kept.

In that study, the same questions went to two model families, and a cross-check decided what reached the final report.

In the September 2026 study, two agents from different model families worked separately, and a cross-check reconciled their reports.

Open diagram at full size

Read the workflow as text

• The research questions went out as parallel queries by separate agents.
• An agent from model family A and an agent from model family B each worked from its own queries and sources.
• Each agent wrote its own report: report A and report B.
• A cross-check of the two reports compared their findings.
• Where the reports disagreed, the more cautious verdict, skip, went forward. Where they agreed, their findings went straight on.
• Both paths ended in one reconciled report.

This was one study. It shows the method, not a rate of errors caught.

Each model edits files in its own format

An agent changes code through an edit tool, and each vendor documents its own. That matters for a harness that runs models from more than one vendor: in September 2026, our review of record used a model from a different vendor than the builder.

Anthropic’s text editor tool gives Claude a str_replace command. The old text must match exactly, including whitespace and indentation, and should match one location. Anthropic describes the tool as schema-less: its schema is built into Claude’s model.

OpenAI’s apply_patch tool lets GPT-5.1, GPT-5.2, GPT-5.4 and GPT-5.5 create, update and delete files with structured diffs in the V4A format. OpenAI’s GPT-4.1 prompting guide publishes a recommended diff format and says the model has been extensively trained on it. The open-source Codex CLI tells its model to try apply_patch for single-file edits.

Google’s Gemini CLI has a replace tool, shown as Edit. It swaps an exact old_string for a new_string and changes one occurrence by default.

My view

Two vendors tie the format to the model: Anthropic builds the edit schema into Claude, and OpenAI trained GPT-4.1 extensively on its diff format. My view is that a model edits most reliably in the format its own harness uses. A harness that serves several models should give each one its native edit tool, not one shared format.

This is a default, not a rule. OpenAI also reports that GPT-4.1 does well with other diff formats given clear instructions and examples.

Sources

• Anthropic: text editor tool
• OpenAI: apply_patch tool
• OpenAI: GPT-4.1 prompting guide
• OpenAI Codex CLI: model prompt
• Google Gemini CLI: file system tools

Public documentation, read on October 2, 2026.

Historical results

This describes the implementation as it stood in September 2026. The boundary suite recorded 87 cases and the build-discipline suite 24 cases, each later reaching zero recorded failures. Those records date from September 2026.

Engineering case study

A task brief only helps if the execution path respects it. I worked with agent teams to turn file scope, review ownership, and approval freshness into checks that could reject an invalid change.

Work: September 6, 2026

The boundary problem

Parallel agent work creates practical questions: which files can this worker change, who owns each part of the diff, and does the approval still apply to the current commit? Leaving those answers in prose makes them easy to misread or carry forward after the work changes. The September 6 work focused on making those boundaries explicit and checkable.

How the checks fit together

A written task brief and file manifest defined the allowed work. File paths were treated as fixed strings rather than loose patterns. Review approval was bound to a commit, so a later change invalidated a stale approval. Ownership checks rejected overlapping claims, missing ownership, and claims for files outside the diff.

Several contracts had to agree

The checks had to connect a task-level promise to a concrete change. The brief declared scope; the manifest named files; the diff showed what changed; ownership assigned responsibility; and review referred to a specific commit. Those records could disagree independently. Complete ownership would still be insufficient if two workers claimed the same file, and a clean ownership ledger would not rescue an approval for an older commit.

A small path bug with a clear consequence

One concrete defect appeared when a requested new path had a parent directory that did not exist. The Git membership check could exit successfully even though that result did not establish the intended boundary. The correction walked to the nearest existing ancestor before checking membership. That detail matters because creating a new file is a normal development action; the control has to reason about a path before the whole path exists.

What I avoided relying on

A permissive path check and a remembered approval were too weak for this workflow. The approach tied permissions to explicit files and review evidence to the actual commit. That adds checks at the cost of friction. The record supports the behavior of those checks, but does not quantify whether the extra steps made the team faster.

What the historical tests showed

The staged boundary suite recorded 87 cases, initially with 51 failures, and later with zero failures and zero skips. A separate build-discipline suite recorded 24 cases, initially with 11 failures, and later with zero failures. These are two different test groups. I keep their counts separate rather than combining them into a single improvement statistic.

What those numbers mean

The result shows that the recorded implementation satisfied the recorded cases after the fixes. It does not establish 51 security vulnerabilities: some failing cases concerned fixture and ledger support. Nor does a zero-failure test report show that every possible path escape, ownership conflict, or stale-state condition has been covered. The value is in the specific rejected behaviors and the repeatable contract they define.

What the evidence covers

The evidence is the September 6, 2026 report. It records the results for the files as they were on that date.

The engineering lesson

I would describe this work as boundary enforcement with documented test evidence. The strongest example is the nonexistent-parent bug: an apparently successful command was answering a weaker question than the workflow required. Naming the exact precondition made the correction concrete.

Learnings

What I learned: a successful command can check a weaker condition than the workflow needs. A new file’s nonexistent parent exposed that problem in the repository-membership check.

What changed: in the September 6, 2026 record, the check walked to the nearest existing ancestor. Ownership checks rejected ambiguous claims, and review approval was bound to the commit.

What I carry forward: prove each control where it runs, not only in a test, and date every test result.