When green tests judge the wrong version

Which test version gets to run?

Before

Candidate introduces a changed test
Trusted copy replaces it before evaluation
The new test is hidden; a broken trusted test can later block repair

After

Run candidate changed tests first
Trusted copies no longer hide the new test
Whether the new test works and whether trusted checks pass stay separate questions

Every tested variant rejected

Saved September 30 run against deliberately altered code. No infrastructure errors.

Scope: one of three evaluation steps. The run had no changed test files, so it is not an end-to-end replay of the changed-test incident.

A test could pass in a normal checkout and fail when the checkout itself lived under a temporary directory. The evaluator’s handling of test versions made that defect harder to see and then harder to repair.

How the failure surfaced

The September 27 defect involved an assertion about temporary output. It behaved as expected in the usual checkout, but failed when the entire checkout was copied under a temporary directory. That environmental difference exposed an assumption in the assertion. The larger problem was which version of the test the evaluation process was actually running.

Why an earlier green result was misleading

The evaluator copied in an older trusted version of the tests. That copy hid a newly changed test, so the defect surfaced on the next pull request instead of the change that introduced it. Once the broken test had become the trusted version, it could block the repair. The workflow had created a loop: a new test could escape evaluation, while its later correction could be judged by the broken predecessor.

Workflow

An older trusted evaluator could hide a candidate test. Once the broken test became trusted, it could block its own correction. — Before the repair

The candidate’s changed tests now run before trusted copies are installed. This shows the ordering change, not a complete CI success path. — After the repair

Two kinds of trust were in conflict

The evaluator had a legitimate reason to preserve trusted tests: code changes should not get an easier grade merely by changing the tests beside them. But copying those tests too early could erase evidence of whether the proposed test itself worked. The temporary-directory failure made that conflict concrete. We needed to evaluate the candidate test before replacement while retaining a check the candidate could not casually rewrite.

The repair I worked on with agent teams

We changed the order so candidate changed tests ran before trusted copies were installed. That separated two useful questions: whether the proposed tests work in the candidate, and whether the code still satisfies independent trusted checks.

This shows the ordering change, not a complete CI success path. — The order of test versions after the repair

Why not take the easy shortcut

Allowing every candidate to replace its own evaluator would weaken the check it was meant to pass. Always restoring the older evaluator preserved the repair deadlock. Neither shortcut kept the comparison meaningful, so the work changed the order of evaluation instead. The implementation record includes changes on September 28 and September 30; those dates describe the work, not a claim that every downstream workflow had been validated.

A concrete saved evaluation

A saved run from September 30 tested the evaluation against deliberately altered variants of the code. It rejected every variant it was given and reported no infrastructure errors. Some of those variants targeted how candidate and trusted tests are compared.

The important restriction on that result

This run covered only one of three evaluation steps. It is not evidence of a full CI pass. It also had no changed test files, so it cannot serve as a before-and-after replay of the original changed-test bug. Those restrictions belong beside the result, because otherwise a clean run can sound like much broader proof than it supplies.

What I can claim

The evidence supports a concrete change to evaluation order and targeted tests of the relevant paths. It does not establish a lower real-world failure rate, nor does it show that the original temporary-directory scenario was replayed end to end in this saved run. I would want that replay, with the test versions recorded at each stage, before treating the incident as fully closed by experimental evidence. Fixing an evaluator has to keep the comparison meaningful; otherwise the fix changes what a green result means.

What I learned

A green result needs an identity: which code, which tests, which baseline, and which environment produced it? Here the test-copy step could silently change the thing being evaluated. Making that order visible was as important as repairing the assertion. The general lesson I take into agent-assisted engineering is to keep independent checks independent and every comparison meaningful, including when the evaluator itself is what needs fixing.